00000002:00010000:1.0:1788201434.086557:0:40406:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e0afd lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbb754 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201434.086586:0:40406:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e0afd lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbb754 expref: -99 pid: 40406 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.086597:0:40406:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/37 (0 0) 00000080:00200000:1.0:1788201434.086604:0:40406:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201434.086645:0:40406:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x25:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201434.086655:0:40406:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201434.086681:0:40406:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x25:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201434.087826:0:40406:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x25:0x0](ffff8e82927cdb88): rc = 0 00000080:00200000:1.0:1788201434.087861:0:40406:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x25:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:1.0:1788201434.087868:0:40406:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x25:0x0](ffff8e82927cdb88) for lock 0x9ae3109cee9e0afd 00000080:00010000:1.0:1788201434.087879:0:40406:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x25:0x0](ffff8e82927cdb88) 00000080:00010000:1.0:1788201434.087883:0:40406:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e0afd from it ffffaa1801973e00 00010000:00010000:1.0:1788201434.087888:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e0afd lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbb754 expref: -99 pid: 40406 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.087898:0:40406:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e0afd lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbb754 expref: -99 pid: 40406 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201434.088637:0:40406:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201434.088660:0:40406:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201434.088676:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f99000/0x9ae3109cee9e0af6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbb738 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201434.088689:0:40406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f99000/0x9ae3109cee9e0af6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbb738 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201434.088715:0:40406:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201434.088721:0:40406:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201434.088725:0:40406:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e0af6 00000080:00010000:1.0:1788201434.088730:0:40406:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e0af6 from it ffffaa1801973d70 00010000:00010000:1.0:1788201434.089119:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f99000/0x9ae3109cee9e0af6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbb738 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201434.089132:0:40406:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f99000/0x9ae3109cee9e0af6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbb738 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201434.089144:0:40406:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201434.089152:0:40406:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11a.replay-vbr-1-23, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201434.089183:0:40406:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f99000/0x9ae3109cee9e0af6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbb738 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201434.089209:0:40406:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e0afd lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cbb754 expref: -99 pid: 40406 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201434.089219:0:40406:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x25:0x0]: losing layout lock 00010000:00010000:1.0:1788201434.089854:0:40406:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e0afd lrc: 1/0,0 mode: --/PR res: [0x200006991:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbb754 expref: -99 pid: 40406 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201434.089865:0:40406:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f99000/0x9ae3109cee9e0af6 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbb738 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201434.089871:0:40406:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201434.089874:0:40406:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e0afd lrc: 0/0,0 mode: --/PR res: [0x200006991:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbb754 expref: -99 pid: 40406 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201434.089901:0:40406:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f99000/0x9ae3109cee9e0af6 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbb738 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201434.089994:0:40406:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201434.090016:0:40406:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290f18e00 x1875062187275392/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201434.090042:0:40406:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f18e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187275392:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201434.097111:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8284fb29c0 going to sleep for 11 seconds 00000080:00200000:1.0:1788201434.177612:0:40406:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x27:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201434.177624:0:40406:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201434.177651:0:40406:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x27:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201434.177722:0:40406:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x27:0x0](ffff8e82927ccb08): rc = 0 00000080:00200000:1.0:1788201434.177740:0:40406:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x27:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:1.0:1788201434.177752:0:40406:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x27:0x0](ffff8e82927ccb08) for lock 0x9ae3109cee9e0b19 00000080:00010000:1.0:1788201434.177763:0:40406:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x27:0x0](ffff8e82927ccb08) 00000080:00010000:1.0:1788201434.177766:0:40406:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e0b19 from it ffffaa1801973e00 00010000:00010000:1.0:1788201434.177770:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e0b19 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x27:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbb7e0 expref: -99 pid: 40406 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.177780:0:40406:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e0b19 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x27:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbb7e0 expref: -99 pid: 40406 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201434.177811:0:40406:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201434.177831:0:40406:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201434.177840:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1e00/0x9ae3109cee9e0b12 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbb7c4 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201434.177858:0:40406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1e00/0x9ae3109cee9e0b12 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbb7c4 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201434.177865:0:40406:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201434.177869:0:40406:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201434.177872:0:40406:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e0b12 00000080:00010000:1.0:1788201434.177877:0:40406:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e0b12 from it ffffaa1801973d70 00010000:00010000:1.0:1788201434.177880:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1e00/0x9ae3109cee9e0b12 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbb7c4 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201434.177887:0:40406:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1e00/0x9ae3109cee9e0b12 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbb7c4 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201434.177896:0:40406:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201434.177904:0:40406:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11a.replay-vbr-1-25, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201434.177931:0:40406:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1e00/0x9ae3109cee9e0b12 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbb7c4 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201434.177954:0:40406:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e0b19 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x27:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cbb7e0 expref: -99 pid: 40406 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201434.177964:0:40406:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x27:0x0]: losing layout lock 00010000:00010000:1.0:1788201434.177995:0:40406:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e0b19 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x27:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbb7e0 expref: -99 pid: 40406 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201434.178015:0:40406:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1e00/0x9ae3109cee9e0b12 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbb7c4 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201434.178033:0:40406:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201434.178036:0:40406:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e0b19 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x27:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbb7e0 expref: -99 pid: 40406 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201434.178053:0:40406:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1e00/0x9ae3109cee9e0b12 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbb7c4 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201434.178073:0:40406:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201434.178082:0:40406:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290f1b800 x1875062187276416/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201434.178105:0:40406:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1b800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187276416:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201434.178233:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8284fb2000 going to sleep for 11 seconds 00000100:00100000:1.0:1788201434.221898:0:40406:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d63d500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187277056:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201434.221930:0:40406:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f98800/0x9ae3109cee9e0b2e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbb850 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.221943:0:40406:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f98800/0x9ae3109cee9e0b2e lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbb850 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.221956:0:40406:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f98800/0x9ae3109cee9e0b2e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbb850 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.221967:0:40406:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f98800/0x9ae3109cee9e0b2e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbb850 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.222000:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f98800/0x9ae3109cee9e0b2e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbb850 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.222020:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f98800/0x9ae3109cee9e0b2e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbb850 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.222029:0:40406:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f98800/0x9ae3109cee9e0b2e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbb850 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201434.222041:0:40406:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d63d500 x1875062187277056/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201450 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201434.222056:0:40406:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f98800/0x9ae3109cee9e0b2e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbb850 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.222068:0:40406:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201434.222076:0:40406:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201434.222101:0:40406:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201434.222107:0:40406:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e0b2e 00000080:00010000:1.0:1788201434.222112:0:40406:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201434.222116:0:40406:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e0b2e from it ffffaa1801973c10 00010000:00010000:1.0:1788201434.222120:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f98800/0x9ae3109cee9e0b2e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbb850 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.222129:0:40406:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f98800/0x9ae3109cee9e0b2e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbb850 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201434.222166:0:40406:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201434.222195:0:40406:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11a.replay-vbr-1-27, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201434.222200:0:40406:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11a.replay-vbr-1-27, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201434.222220:0:40406:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11a.replay-vbr-1-27,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201434.222262:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6c00/0x9ae3109cee9e0b35 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.222270:0:40406:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6c00/0x9ae3109cee9e0b35 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.222280:0:40406:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6c00/0x9ae3109cee9e0b35 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201434.222299:0:40406:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1bb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187277184:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201434.304669:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82878a3a80 going to sleep for 11 seconds 00010000:00010000:1.0:1788201434.317558:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201434.492847:0:40406:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d634a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187285120:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201434.492912:0:40406:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x38:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201434.492925:0:40406:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201434.492955:0:40406:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x38:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201434.492963:0:40406:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x38:0x0](ffff8e82927dd348) 00000080:00200000:1.0:1788201434.493068:0:40406:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201434.493092:0:40406:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201434.493154:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0c0e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.493165:0:40406:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0c0e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.493189:0:40406:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0c0e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201434.493210:0:40406:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f19500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187285248:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201434.493339:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8284fb23c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201434.498352:0:40406:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f19500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187285248:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201434.498380:0:40406:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0c0e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbcb0 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.498392:0:40406:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0c0e lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbcb0 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.498402:0:40406:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0c0e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbcb0 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.498410:0:40406:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0c0e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbcb0 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.498420:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0c0e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbcb0 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.498425:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0c0e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbcb0 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.498429:0:40406:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0c0e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbcb0 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201434.498457:0:40406:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290f19500 x1875062187285248/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201450 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201434.498468:0:40406:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0c0e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbcb0 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.498478:0:40406:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201434.498483:0:40406:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201434.498504:0:40406:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201434.498509:0:40406:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e0c0e 00000080:00010000:1.0:1788201434.498512:0:40406:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201434.498514:0:40406:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e0c0e from it ffffaa1801973c10 00010000:00010000:1.0:1788201434.498516:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0c0e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbcb0 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.498522:0:40406:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0c0e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbcb0 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201434.498542:0:40406:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201434.498568:0:40406:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11a.replay-vbr-1-43, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201434.498572:0:40406:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11a.replay-vbr-1-43, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201434.498589:0:40406:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11a.replay-vbr-1-43,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201434.498617:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6400/0x9ae3109cee9e0c15 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.498621:0:40406:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6400/0x9ae3109cee9e0c15 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.498636:0:40406:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6400/0x9ae3109cee9e0c15 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201434.498651:0:40406:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f19500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187285376:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201434.498721:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8284fb23c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201434.502092:0:40406:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f19500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187285376:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201434.502123:0:40406:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6400/0x9ae3109cee9e0c15 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbccc expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.502144:0:40406:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6400/0x9ae3109cee9e0c15 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x39:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbccc expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.502152:0:40406:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6400/0x9ae3109cee9e0c15 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbccc expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.502163:0:40406:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6400/0x9ae3109cee9e0c15 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbccc expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.502174:0:40406:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6400/0x9ae3109cee9e0c15 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbccc expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.502184:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6400/0x9ae3109cee9e0c15 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbccc expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.502190:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6400/0x9ae3109cee9e0c15 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbccc expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.502207:0:40406:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6400/0x9ae3109cee9e0c15 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbccc expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201434.502216:0:40406:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290f19500 x1875062187285376/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201450 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201434.502234:0:40406:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6400/0x9ae3109cee9e0c15 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbccc expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201434.502244:0:40406:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6400/0x9ae3109cee9e0c15 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbccc expref: -99 pid: 40406 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.502252:0:40406:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/57 (0 0) 00000080:00200000:1.0:1788201434.502257:0:40406:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201434.502297:0:40406:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x39:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201434.502305:0:40406:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201434.502352:0:40406:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x39:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:1.0:1788201434.535377:0:40406:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d637800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187286272:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201434.535406:0:40406:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639400/0x9ae3109cee9e0c2a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbd3c expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.535419:0:40406:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639400/0x9ae3109cee9e0c2a lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbd3c expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.535430:0:40406:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639400/0x9ae3109cee9e0c2a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbd3c expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.535460:0:40406:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639400/0x9ae3109cee9e0c2a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbd3c expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.535475:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639400/0x9ae3109cee9e0c2a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbd3c expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.535484:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639400/0x9ae3109cee9e0c2a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbd3c expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.535492:0:40406:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639400/0x9ae3109cee9e0c2a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbd3c expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201434.535506:0:40406:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d637800 x1875062187286272/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201450 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201434.535522:0:40406:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639400/0x9ae3109cee9e0c2a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbd3c expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.535536:0:40406:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201434.535544:0:40406:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201434.535570:0:40406:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201434.535576:0:40406:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e0c2a 00000080:00010000:1.0:1788201434.535581:0:40406:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201434.535585:0:40406:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e0c2a from it ffffaa1801973c10 00010000:00010000:1.0:1788201434.535589:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639400/0x9ae3109cee9e0c2a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbd3c expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.535609:0:40406:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639400/0x9ae3109cee9e0c2a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbd3c expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201434.535633:0:40406:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201434.535666:0:40406:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11a.replay-vbr-1-45, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201434.535671:0:40406:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11a.replay-vbr-1-45, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201434.535694:0:40406:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11a.replay-vbr-1-45,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201434.535737:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6400/0x9ae3109cee9e0c31 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.535746:0:40406:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6400/0x9ae3109cee9e0c31 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.535756:0:40406:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6400/0x9ae3109cee9e0c31 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201434.535776:0:40406:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1b100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187286400:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201434.535853:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8284fb23c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201434.589159:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82922cac00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201434.781759:0:40406:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187290752:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201434.786742:0:40406:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe1f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187290880:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201434.786769:0:40406:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e0ca8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbfb2 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.786782:0:40406:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e0ca8 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbfb2 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.786807:0:40406:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e0ca8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbfb2 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.786820:0:40406:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e0ca8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbfb2 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.786836:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e0ca8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbfb2 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.786845:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e0ca8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbfb2 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.786854:0:40406:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e0ca8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbfb2 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201434.786867:0:40406:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290fe1f80 x1875062187290880/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201450 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201434.786882:0:40406:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e0ca8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbfb2 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.786896:0:40406:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201434.786903:0:40406:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201434.786928:0:40406:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201434.786933:0:40406:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e0ca8 00000080:00010000:1.0:1788201434.786939:0:40406:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201434.786942:0:40406:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e0ca8 from it ffffaa1801973c10 00010000:00010000:1.0:1788201434.786963:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e0ca8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbfb2 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.786974:0:40406:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e0ca8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbfb2 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201434.787016:0:40406:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201434.787049:0:40406:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11a.replay-vbr-1-54, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201434.787054:0:40406:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11a.replay-vbr-1-54, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201434.787073:0:40406:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11a.replay-vbr-1-54,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201434.787114:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0caf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.787122:0:40406:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0caf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.787131:0:40406:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0caf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201434.787151:0:40406:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f19c00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187291008:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201434.787265:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8284fb2a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201434.794679:0:40406:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f19c00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187291008:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201434.794708:0:40406:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0caf lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbfce expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.794730:0:40406:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0caf lrc: 4/1,0 mode: --/PR res: [0x200006991:0x44:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbfce expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.794754:0:40406:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0caf lrc: 4/1,0 mode: --/PR res: [0x200006991:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbfce expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.794765:0:40406:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0caf lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbfce expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.794776:0:40406:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0caf lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbfce expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.794789:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0caf lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbfce expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.794798:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0caf lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbfce expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.794806:0:40406:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0caf lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbfce expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201434.794818:0:40406:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290f19c00 x1875062187291008/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201450 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201434.794831:0:40406:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0caf lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbfce expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201434.794844:0:40406:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0caf lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbfce expref: -99 pid: 40406 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.794865:0:40406:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/68 (0 0) 00000080:00200000:1.0:1788201434.794872:0:40406:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201434.794911:0:40406:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x44:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201434.794922:0:40406:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201434.794986:0:40406:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x44:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201434.795064:0:40406:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x44:0x0](ffff8e82927f21c8): rc = 0 00000080:00200000:1.0:1788201434.795078:0:40406:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x44:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:1.0:1788201434.795085:0:40406:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x44:0x0](ffff8e82927f21c8) for lock 0x9ae3109cee9e0caf 00000080:00010000:1.0:1788201434.795094:0:40406:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x44:0x0](ffff8e82927f21c8) 00000080:00010000:1.0:1788201434.795098:0:40406:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e0caf from it ffffaa1801973e00 00010000:00010000:1.0:1788201434.795102:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0caf lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbbfce expref: -99 pid: 40406 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.795112:0:40406:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0caf lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbbfce expref: -99 pid: 40406 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201434.795146:0:40406:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201434.795162:0:40406:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201434.795172:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e0ca8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbfb2 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201434.795182:0:40406:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e0ca8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbfb2 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201434.795205:0:40406:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201434.795210:0:40406:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201434.795220:0:40406:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e0ca8 00000080:00010000:1.0:1788201434.795224:0:40406:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e0ca8 from it ffffaa1801973d70 00010000:00010000:1.0:1788201434.795228:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e0ca8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbbfb2 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201434.795235:0:40406:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e0ca8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbbfb2 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201434.795245:0:40406:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201434.795252:0:40406:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11a.replay-vbr-1-54, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201434.795278:0:40406:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e0ca8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbbfb2 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201434.795300:0:40406:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0caf lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cbbfce expref: -99 pid: 40406 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201434.795364:0:40406:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x44:0x0]: losing layout lock 00010000:00010000:1.0:1788201434.795398:0:40406:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0caf lrc: 1/0,0 mode: --/PR res: [0x200006991:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbbfce expref: -99 pid: 40406 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201434.795406:0:40406:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e0ca8 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbbfb2 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201434.795413:0:40406:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201434.795416:0:40406:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0caf lrc: 0/0,0 mode: --/PR res: [0x200006991:0x44:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbbfce expref: -99 pid: 40406 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201434.795436:0:40406:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e0ca8 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbbfb2 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201434.795473:0:40406:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201434.795481:0:40406:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290f19c00 x1875062187291264/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201434.795515:0:40406:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f19c00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187291264:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201434.795604:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8284fb2b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201434.811741:0:40406:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f19c00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187291264:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201434.811804:0:40406:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x44:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201434.811817:0:40406:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201434.811845:0:40406:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x44:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201434.811853:0:40406:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x44:0x0](ffff8e82927f21c8) 00000080:00200000:1.0:1788201434.811942:0:40406:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201434.811974:0:40406:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201434.812035:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6200/0x9ae3109cee9e0cb6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.812047:0:40406:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6200/0x9ae3109cee9e0cb6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.812056:0:40406:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6200/0x9ae3109cee9e0cb6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201434.812078:0:40406:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f18000 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187291392:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201434.812195:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8284fb2a80 going to sleep for 11 seconds 00010000:00010000:1.0:1788201434.824607:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0cbd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.824651:0:40406:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0cbd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.824667:0:40406:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7c00/0x9ae3109cee9e0cbd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201434.824703:0:40406:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187291520:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201434.824881:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8284fb2b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201434.986787:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d7fc00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201434.986876:0:40406:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe1500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187293312:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201434.987107:0:40406:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x48:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201434.987123:0:40406:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201434.987160:0:40406:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x48:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201434.987169:0:40406:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x48:0x0](ffff8e82927e1148) 00000080:00200000:1.0:1788201434.987269:0:40406:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201434.987294:0:40406:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201434.987387:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6c00/0x9ae3109cee9e0cee lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.987398:0:40406:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6c00/0x9ae3109cee9e0cee lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201434.987410:0:40406:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6c00/0x9ae3109cee9e0cee lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201434.987431:0:40406:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f19f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187293440:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201434.987513:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8284fb2e40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201435.046227:0:40406:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ed80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187293952:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201435.046253:0:40406:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639000/0x9ae3109cee9e0cfc lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbc156 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201435.046265:0:40406:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639000/0x9ae3109cee9e0cfc lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbc156 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201435.046463:0:40406:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639000/0x9ae3109cee9e0cfc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbc156 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201435.046477:0:40406:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639000/0x9ae3109cee9e0cfc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbc156 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201435.046491:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639000/0x9ae3109cee9e0cfc lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbc156 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201435.046501:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639000/0x9ae3109cee9e0cfc lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbc156 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201435.046510:0:40406:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639000/0x9ae3109cee9e0cfc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbc156 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201435.046521:0:40406:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793ed80 x1875062187293952/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201451 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201435.046536:0:40406:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639000/0x9ae3109cee9e0cfc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbc156 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201435.046548:0:40406:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201435.046570:0:40406:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201435.046653:0:40406:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201435.046661:0:40406:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e0cfc 00000080:00010000:1.0:1788201435.046668:0:40406:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201435.046671:0:40406:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e0cfc from it ffffaa1801973c10 00010000:00010000:1.0:1788201435.046675:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639000/0x9ae3109cee9e0cfc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbc156 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201435.046684:0:40406:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639000/0x9ae3109cee9e0cfc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbc156 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201435.046706:0:40406:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201435.046798:0:40406:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11a.replay-vbr-1-60, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201435.046803:0:40406:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11a.replay-vbr-1-60, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201435.046870:0:40406:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11a.replay-vbr-1-60,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201435.046962:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6600/0x9ae3109cee9e0d03 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201435.046970:0:40406:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6600/0x9ae3109cee9e0d03 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201435.046978:0:40406:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6600/0x9ae3109cee9e0d03 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201435.047069:0:40406:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f18380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187294080:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201435.047154:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8284fb2e40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201435.069981:0:40406:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187294336:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201435.070131:0:40406:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x4a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201435.070161:0:40406:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201435.070193:0:40406:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x4a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201435.070204:0:40406:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x4a:0x0](ffff8e82927e80c8) 00000080:00200000:1.0:1788201435.070720:0:40406:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201435.070743:0:40406:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201435.070803:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6400/0x9ae3109cee9e0d0a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201435.070814:0:40406:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6400/0x9ae3109cee9e0d0a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201435.072679:0:40406:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6400/0x9ae3109cee9e0d0a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201435.072712:0:40406:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1ad80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187294464:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201435.072812:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8284fb2b40 going to sleep for 11 seconds 00000080:00200000:1.0:1788201435.132922:0:40406:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201435.132941:0:40406:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e0d26 00000080:00010000:1.0:1788201435.132949:0:40406:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201435.132953:0:40406:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e0d26 from it ffffaa1801973c10 00010000:00010000:1.0:1788201435.132959:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639000/0x9ae3109cee9e0d26 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbc228 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201435.132972:0:40406:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639000/0x9ae3109cee9e0d26 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbc228 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201435.133015:0:40406:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201435.133047:0:40406:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11a.replay-vbr-1-63, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201435.133052:0:40406:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11a.replay-vbr-1-63, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201435.133094:0:40406:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11a.replay-vbr-1-63,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201435.133148:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0d2d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201435.133157:0:40406:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0d2d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201435.133168:0:40406:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0d2d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201435.133194:0:40406:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f19880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187295616:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201435.133713:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8284fb2a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201435.474394:0:40406:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187299968:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201435.474460:0:40406:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x55:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201435.474474:0:40406:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201435.474503:0:40406:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x55:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201435.474511:0:40406:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x55:0x0](ffff8e82927fba88) 00000080:00200000:1.0:1788201435.474606:0:40406:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201435.474629:0:40406:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201435.474684:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6a00/0x9ae3109cee9e0da4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201435.474695:0:40406:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6a00/0x9ae3109cee9e0da4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201435.474704:0:40406:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6a00/0x9ae3109cee9e0da4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201435.474802:0:40406:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1a300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187300096:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201435.475645:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8284fb23c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201435.651841:0:40406:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe3b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187302528:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201435.651919:0:40406:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x5a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201435.651936:0:40406:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201435.652013:0:40406:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x5a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201435.652023:0:40406:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x5a:0x0](ffff8e82927fd348) 00000080:00200000:1.0:1788201435.652139:0:40406:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201435.652166:0:40406:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201435.652238:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7a00/0x9ae3109cee9e0dea lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201435.652249:0:40406:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7a00/0x9ae3109cee9e0dea lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201435.652261:0:40406:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7a00/0x9ae3109cee9e0dea lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201435.652285:0:40406:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f18700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187302656:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201435.652678:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8284fb29c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201435.712135:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287e509c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201435.760527:0:40406:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe0a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187303040:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201436.414477:0:40406:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe1c00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187309696:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201436.414557:0:40406:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x68:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201436.414574:0:40406:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201436.414609:0:40406:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x68:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201436.414619:0:40406:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x68:0x0](ffff8e8289c163c8) 00000080:00200000:1.0:1788201436.414982:0:40406:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201436.415011:0:40406:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201436.415088:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0eae lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.415099:0:40406:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0eae lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.415111:0:40406:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0eae lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201436.415136:0:40406:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1b480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187309824:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201436.415384:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8284fb2600 going to sleep for 11 seconds 00000100:00100000:1.0:1788201436.420601:0:40406:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f1b480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187309824:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201436.420644:0:40406:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0eae lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbc9d0 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.420660:0:40406:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0eae lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbc9d0 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.420685:0:40406:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0eae lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbc9d0 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.420699:0:40406:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0eae lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbc9d0 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.420714:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0eae lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbc9d0 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.420753:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0eae lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbc9d0 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.420763:0:40406:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0eae lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbc9d0 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201436.420776:0:40406:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290f1b480 x1875062187309824/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201452 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201436.420805:0:40406:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0eae lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbc9d0 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.420824:0:40406:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201436.420832:0:40406:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201436.420978:0:40406:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201436.420987:0:40406:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e0eae 00000080:00010000:1.0:1788201436.420994:0:40406:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201436.420998:0:40406:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e0eae from it ffffaa1801973c10 00010000:00010000:1.0:1788201436.421001:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0eae lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbc9d0 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.421011:0:40406:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7600/0x9ae3109cee9e0eae lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbc9d0 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201436.421044:0:40406:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201436.421094:0:40406:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11a.replay-vbr-1-91, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201436.421100:0:40406:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11a.replay-vbr-1-91, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201436.421139:0:40406:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11a.replay-vbr-1-91,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201436.421200:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7a00/0x9ae3109cee9e0eb5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.421208:0:40406:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7a00/0x9ae3109cee9e0eb5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.421218:0:40406:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb7a00/0x9ae3109cee9e0eb5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201436.421243:0:40406:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1b480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187309952:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201436.421370:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8284fb2600 going to sleep for 11 seconds 00000100:00100000:1.0:1788201436.716430:0:40406:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828786d180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187313920:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201436.716468:0:40406:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639600/0x9ae3109cee9e0f1e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbcc00 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.716484:0:40406:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639600/0x9ae3109cee9e0f1e lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbcc00 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.716499:0:40406:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639600/0x9ae3109cee9e0f1e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbcc00 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.716512:0:40406:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639600/0x9ae3109cee9e0f1e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbcc00 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.716528:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639600/0x9ae3109cee9e0f1e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbcc00 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.716537:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639600/0x9ae3109cee9e0f1e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbcc00 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.716568:0:40406:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639600/0x9ae3109cee9e0f1e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbcc00 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201436.716583:0:40406:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828786d180 x1875062187313920/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201452 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201436.716611:0:40406:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639600/0x9ae3109cee9e0f1e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbcc00 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.716630:0:40406:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201436.716640:0:40406:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201436.716680:0:40406:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201436.716689:0:40406:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e0f1e 00000080:00010000:1.0:1788201436.716694:0:40406:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201436.716698:0:40406:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e0f1e from it ffffaa1801973c10 00010000:00010000:1.0:1788201436.716701:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639600/0x9ae3109cee9e0f1e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbcc00 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.716710:0:40406:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828d639600/0x9ae3109cee9e0f1e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbcc00 expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201436.716739:0:40406:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201436.716783:0:40406:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11a.replay-vbr-1-99, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201436.717094:0:40406:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11a.replay-vbr-1-99, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201436.717125:0:40406:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11a.replay-vbr-1-99,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201436.717186:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6c00/0x9ae3109cee9e0f25 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.717203:0:40406:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6c00/0x9ae3109cee9e0f25 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.717215:0:40406:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6c00/0x9ae3109cee9e0f25 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40406 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201436.717243:0:40406:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1b480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187314048:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201436.718700:0:40406:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8284fb2000 going to sleep for 11 seconds 00000100:00100000:1.0:1788201436.723577:0:40406:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f1b480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40406:1875062187314048:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201436.723626:0:40406:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6c00/0x9ae3109cee9e0f25 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbcc1c expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.723655:0:40406:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6c00/0x9ae3109cee9e0f25 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x71:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbcc1c expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.723664:0:40406:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6c00/0x9ae3109cee9e0f25 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x71:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbcc1c expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.723677:0:40406:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6c00/0x9ae3109cee9e0f25 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x71:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbcc1c expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.723691:0:40406:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6c00/0x9ae3109cee9e0f25 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x71:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbcc1c expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.723707:0:40406:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6c00/0x9ae3109cee9e0f25 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x71:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbcc1c expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.723735:0:40406:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6c00/0x9ae3109cee9e0f25 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x71:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbcc1c expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.723743:0:40406:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6c00/0x9ae3109cee9e0f25 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x71:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbcc1c expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201436.723757:0:40406:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290f1b480 x1875062187314048/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201452 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201436.723770:0:40406:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6c00/0x9ae3109cee9e0f25 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x71:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbcc1c expref: -99 pid: 40406 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201436.723785:0:40406:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290eb6c00/0x9ae3109cee9e0f25 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x71:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbcc1c expref: -99 pid: 40406 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201436.723797:0:40406:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/113 (0 0) 00000080:00200000:1.0:1788201436.723806:0:40406:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201436.723871:0:40406:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x71:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201436.723886:0:40406:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201436.723926:0:40406:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x71:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:1.0:1788201438.217630:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187314688:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201438.218519:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187314944:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201438.218859:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187315200:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201438.218892:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788201438.218900:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187314816:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201438.219082:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187315072:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201438.250156:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187314816:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201438.252858:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187315072:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201438.252888:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187315200:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201438.252902:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187314688:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201438.252915:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187314944:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201443.332936:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187315328:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201443.333053:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b92a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187315584:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201443.333085:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187315840:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201443.333113:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788201443.333119:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b90000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187315456:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201443.333143:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b90380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187315712:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201446.461368:0:41001:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f18700 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:41001:1875062187316224:192.168.201.121@tcp:503:mount.lustre.0 00000100:00100000:1.0:1788201446.461801:0:41001:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63e40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201446.483301:0:41001:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f18700 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:41001:1875062187316224:192.168.201.121@tcp:503:mount.lustre.0 00000100:00100000:1.0:1788201446.494350:0:41009:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f18700 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:41009:1875062187316352:192.168.201.121@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788201446.494560:0:41009:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201446.497006:0:41009:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f18700 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:41009:1875062187316352:192.168.201.121@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788201446.497051:0:41009:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788201446.497063:0:41009:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788201446.497065:0:41009:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) lustre-clilov lov setup 00000020:01000000:1.0:1788201446.497072:0:41009:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilov-ffff8e8284ef2800 00000020:00000080:1.0:1788201446.497074:0:41009:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:1.0:1788201446.497592:0:41009:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilov-ffff8e8284ef2800 (ffff8e8283ee19b8) 00000020:00000080:1.0:1788201446.497615:0:41009:0:(obd_config.c:720:class_attach_name()) OBD: dev 7 attached type lov with refcount 2 00000020:01000000:1.0:1788201446.497625:0:41009:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilov-ffff8e8284ef2800 00000020:00000080:1.0:1788201446.497629:0:41009:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000100:00100000:1.0:1788201446.520274:0:41009:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290efd500 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:41009:1875062187316480:192.168.201.121@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788201446.520341:0:41009:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788201446.520351:0:41009:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788201446.520353:0:41009:0:(obd_config.c:1563:class_process_config()) marker 42 (0x1) lustre sys.jobid_var 00000020:00000080:1.0:1788201446.520360:0:41009:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00f 00000100:00100000:1.0:1788201446.520398:0:41009:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f18700 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:41009:1875062187316608:192.168.201.121@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788201446.520473:0:41009:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a633c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201446.525683:0:41009:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f18700 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:41009:1875062187316608:192.168.201.121@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788201446.525729:0:41009:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788201446.525739:0:41009:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788201446.525742:0:41009:0:(obd_config.c:1563:class_process_config()) marker 42 (0x2) lustre sys.jobid_var 00000040:00080000:1.0:1788201446.525747:0:41009:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 52 count 51 00000020:01000000:1.0:1788201446.561196:0:41001:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 1-6 (rc=0) 10000000:01000000:1.0:1788201446.561232:0:41001:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.121@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:1.0:1788201446.561259:0:41001:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e828d639400/0x9ae3109cee9e0f72 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cbcf2c expref: -99 pid: 41001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201446.561272:0:41001:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.121@tcp lock: ffff8e828d639400/0x9ae3109cee9e0f72 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cbcf2c expref: -99 pid: 41001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:01000000:1.0:1788201446.561301:0:41001:0:(llite_lib.c:1614:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000020:00000080:1.0:1788201446.561342:0:41001:0:(genops.c:1385:class_connect()) connect: client 6fe158f2-55b2-4a06-bd78-c92321304d95, cookie 0x9ae3109cee9e0f79 00800000:01000000:1.0:1788201446.561363:0:41001:0:(lmv_obd.c:515:lmv_check_connect()) Time to connect 6fe158f2-55b2-4a06-bd78-c92321304d95 to lustre-clilmv-ffff8e8284ef2800 00800000:01000000:1.0:1788201446.561372:0:41001:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0000-mdc-ffff8e8284ef2800(6fe158f2-55b2-4a06-bd78-c92321304d95) - lustre-MDT0000_UUID, 6fe158f2-55b2-4a06-bd78-c92321304d95 00000020:00000080:1.0:1788201446.561379:0:41001:0:(genops.c:1385:class_connect()) connect: client 6fe158f2-55b2-4a06-bd78-c92321304d95, cookie 0x9ae3109cee9e0f80 00000100:00080000:1.0:1788201446.561389:0:41001:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8284ef2800: connect to NID 192.168.201.121@tcp last attempt 0 00000100:00080000:1.0:1788201446.561400:0:41001:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8284ef2800: import ffff8e82911cb800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201446.561411:0:41001:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1788201446.561436:0:41001:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8290f18700 x1875062187317632/t0(0) o38->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201446.561456:0:41001:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 6fe158f2-55b2-4a06-bd78-c92321304d95->lustre-MDT0000_UUID 00000100:00100000:1.0:1788201446.567304:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:1.0:1788201446.567356:0:2401:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8e8288b91880 x1875062187318016/t0(0) o41->lustre-MDT0001-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201446.583994:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6fe158f2-55b2-4a06-bd78-c92321304d95:0:1875062187318016:192.168.201.121@tcp:41:mount.lustre.0 00000100:00100000:1.0:1788201446.584023:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6fe158f2-55b2-4a06-bd78-c92321304d95:0:1875062187317760:192.168.201.121@tcp:41:mount.lustre.0 00000100:00100000:1.0:1788201446.585482:0:41001:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b90380 pname:cluuid:pid:xid:nid:opc:job mount.lustre:6fe158f2-55b2-4a06-bd78-c92321304d95:41001:1875062187318144:192.168.201.121@tcp:41:mount.lustre.0 00800000:00000004:1.0:1788201446.585506:0:41001:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8e8284ef2800: update ffff8e8283e9b7c0 cache blocks 317376/353529 objects 1023689/1024000 00000080:00000004:1.0:1788201446.585513:0:41001:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff8e8284ef2800: update ffff8e8283e9d178 cache blocks 317376/353529 objects 1023689/1024000 00000100:00100000:1.0:1788201446.585552:0:41001:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1a300 pname:cluuid:pid:xid:nid:opc:job mount.lustre:6fe158f2-55b2-4a06-bd78-c92321304d95:41001:1875062187318784:192.168.201.121@tcp:40:mount.lustre.0 00000100:00100000:1.0:1788201446.585741:0:41001:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a630c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201447.686955:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.121@tcp (source: 12345-192.168.201.121@tcp) 00000080:00200000:1.0:1788201448.549008:0:41043:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201448.549017:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e11fd 00000080:00010000:1.0:1788201448.549023:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.549027:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e11fd from it ffffaa1800c97bf8 00010000:00010000:1.0:1788201448.549030:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee0600/0x9ae3109cee9e11fd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd791 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.549040:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee0600/0x9ae3109cee9e11fd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd791 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.549069:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201448.549103:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.549122:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.549133:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee0600/0x9ae3109cee9e11fd lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd791 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.549141:0:41043:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee0600/0x9ae3109cee9e11fd lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd791 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.549147:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201448.549152:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.549155:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e11fd 00000080:00010000:1.0:1788201448.549159:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e11fd from it ffffaa1800c97b38 00010000:00010000:1.0:1788201448.549161:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee0600/0x9ae3109cee9e11fd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd791 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.549181:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee0600/0x9ae3109cee9e11fd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd791 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201448.549188:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201448.549194:0:41043:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-2-43, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), file ffff8e829186be00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201448.549203:0:41043:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-2-43, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000002:00010000:1.0:1788201448.549222:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-2-43,[0x2000088d1:0x2c:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201448.549236:0:41043:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee0600/0x9ae3109cee9e11fd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbd791 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.549284:0:41043:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee0600/0x9ae3109cee9e11fd lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbd791 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.549292:0:41043:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201448.549294:0:41043:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee0600/0x9ae3109cee9e11fd lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbd791 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.549364:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6200/0x9ae3109cee9e1204 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.549371:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6200/0x9ae3109cee9e1204 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201448.549380:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201448.549387:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290f1ad80 x1875062187347584/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201448.549396:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6200/0x9ae3109cee9e1204 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.549422:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1ad80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187347584:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.549535:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.612471:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7b80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187350144:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.612496:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1000/0x9ae3109cee9e123c lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd871 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.612525:0:41043:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1000/0x9ae3109cee9e123c lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x30:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd871 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.612533:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1000/0x9ae3109cee9e123c lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd871 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.612544:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1000/0x9ae3109cee9e123c lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd871 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.612554:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1000/0x9ae3109cee9e123c lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd871 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.612567:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1000/0x9ae3109cee9e123c lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd871 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.612575:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1000/0x9ae3109cee9e123c lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd871 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.612582:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1000/0x9ae3109cee9e123c lrc: 4/1,0 mode: PR/PR res: [0x2000088d1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd871 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.612623:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e82894e7b80 x1875062187350144/t77309411523(77309411523) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201448.612636:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e82894e7b80 x1875062187350144/t77309411523(77309411523) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.612650:0:41043:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1000/0x9ae3109cee9e123c lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd871 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201448.612673:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1000/0x9ae3109cee9e123c lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd871 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.612685:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589969617/48 (0 0) 00000080:00200000:1.0:1788201448.612691:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201448.612705:0:41043:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-2-47, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000080:00200000:1.0:1788201448.612733:0:41043:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000088d1:0x30:0x0](ffff8e82927f21c8) 00000080:00200000:1.0:1788201448.612954:0:41043:0:(file.c:6843:ll_layout_conf()) [0x2000088d1:0x30:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201448.612959:0:41043:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2000088d1:0x30:0x0](ffff8e82927f21c8) 00000080:00010000:1.0:1788201448.612963:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000088d1:0x30:0x0](ffff8e82927f21c8) for lock 0x9ae3109cee9e123c 00000080:00010000:1.0:1788201448.612973:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000088d1:0x30:0x0](ffff8e82927f21c8) 00000080:00200008:1.0:1788201448.612987:0:41043:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-2-47:[0x2000088d1:0x30:0x0](ffff8e82927f21c8), flags 100102 00000002:00100000:1.0:1788201448.613009:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e82894e7b80 x1875062187350144/t77309411523(77309411523) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201448.613040:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e123c from it ffff8e829087e5a0 00010000:00010000:1.0:1788201448.613044:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1000/0x9ae3109cee9e123c lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbd871 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.613062:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1000/0x9ae3109cee9e123c lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbd871 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.613082:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000088d1:0x30:0x0](ffff8e82927f21c8), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201448.613113:0:41043:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-2-47:[0x2000088d1:0x30:0x0](ffff8e82927f21c8), flags 100002 00000002:00100000:1.0:1788201448.613155:0:41043:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e82894e7b80 x1875062187350144/t77309411523(77309411523) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201448.613164:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201448.613168:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290f18380 x1875062187350400/t0(0) o35->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201448.613190:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f18380 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187350400:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201448.613271:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a633c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.617088:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f18380 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187350400:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201448.617123:0:41043:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e82894e7b80 x1875062187350144/t77309411523(77309411523) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201448.617201:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.617218:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.617279:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1243 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.617289:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1243 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.617300:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1243 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.617347:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f19f80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187350528:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.617411:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a633c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.620645:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f19f80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187350528:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.620670:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1243 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd886 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.620682:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1243 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd886 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.620693:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1243 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd886 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.620703:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1243 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd886 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.620714:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1243 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd886 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.620722:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1243 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd886 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.620729:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1243 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd886 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.620740:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290f19f80 x1875062187350528/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.620793:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1243 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd886 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.620806:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201448.620815:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201448.620842:0:41043:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201448.620848:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e1243 00000080:00010000:1.0:1788201448.620853:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.620856:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1243 from it ffffaa1800c97bf8 00010000:00010000:1.0:1788201448.620859:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1243 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd886 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.620868:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1243 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd886 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.620892:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201448.620914:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.620928:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.620936:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1243 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd886 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.620946:0:41043:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1243 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd886 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.620953:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201448.620957:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.620960:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e1243 00000080:00010000:1.0:1788201448.620970:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1243 from it ffffaa1800c97b38 00010000:00010000:1.0:1788201448.620973:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1243 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd886 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.620980:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1243 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd886 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201448.620989:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201448.621005:0:41043:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-2-48, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), file ffff8e82829b9380, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201448.621014:0:41043:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-2-48, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000002:00010000:1.0:1788201448.621035:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-2-48,[0x2000088d1:0x31:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201448.621046:0:41043:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1243 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbd886 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.621096:0:41043:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1243 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbd886 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.621103:0:41043:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201448.621106:0:41043:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1243 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbd886 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.621139:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7400/0x9ae3109cee9e124a lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.621145:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7400/0x9ae3109cee9e124a lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201448.621154:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201448.621161:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290f19f80 x1875062187350784/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201448.621175:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7400/0x9ae3109cee9e124a lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.621194:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f19f80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187350784:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.621346:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a633c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.625880:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f19f80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187350784:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.625907:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7400/0x9ae3109cee9e124a lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8a2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.625935:0:41043:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7400/0x9ae3109cee9e124a lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x31:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8a2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.625943:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7400/0x9ae3109cee9e124a lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8a2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.625954:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7400/0x9ae3109cee9e124a lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd8a2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.625964:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7400/0x9ae3109cee9e124a lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8a2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.625976:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7400/0x9ae3109cee9e124a lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd8a2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.625983:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7400/0x9ae3109cee9e124a lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd8a2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.626022:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7400/0x9ae3109cee9e124a lrc: 4/1,0 mode: PR/PR res: [0x2000088d1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd8a2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.626034:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8290f19f80 x1875062187350784/t77309411525(77309411525) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201448.626047:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290f19f80 x1875062187350784/t77309411525(77309411525) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.626060:0:41043:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7400/0x9ae3109cee9e124a lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8a2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201448.626073:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7400/0x9ae3109cee9e124a lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8a2 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.626084:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589969617/49 (0 0) 00000080:00200000:1.0:1788201448.626089:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201448.626102:0:41043:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-2-48, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000080:00200000:1.0:1788201448.626133:0:41043:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000088d1:0x31:0x0](ffff8e82927f5348) 00000080:00200000:1.0:1788201448.626275:0:41043:0:(file.c:6843:ll_layout_conf()) [0x2000088d1:0x31:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201448.626280:0:41043:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2000088d1:0x31:0x0](ffff8e82927f5348) 00000080:00010000:1.0:1788201448.626283:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000088d1:0x31:0x0](ffff8e82927f5348) for lock 0x9ae3109cee9e124a 00000080:00010000:1.0:1788201448.626292:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000088d1:0x31:0x0](ffff8e82927f5348) 00000080:00200008:1.0:1788201448.626336:0:41043:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-2-48:[0x2000088d1:0x31:0x0](ffff8e82927f5348), flags 100102 00000002:00100000:1.0:1788201448.626357:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290f19f80 x1875062187350784/t77309411525(77309411525) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201448.626397:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e124a from it ffff8e8287377240 00010000:00010000:1.0:1788201448.626402:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7400/0x9ae3109cee9e124a lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbd8a2 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.626411:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7400/0x9ae3109cee9e124a lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbd8a2 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.626430:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000088d1:0x31:0x0](ffff8e82927f5348), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201448.626456:0:41043:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-2-48:[0x2000088d1:0x31:0x0](ffff8e82927f5348), flags 100002 00000002:00100000:1.0:1788201448.626493:0:41043:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8290f19f80 x1875062187350784/t77309411525(77309411525) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201448.626502:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201448.626507:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290f18000 x1875062187351040/t0(0) o35->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201448.626528:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f18000 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187351040:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201448.626776:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63e40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.629783:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f18000 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187351040:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201448.629818:0:41043:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8290f19f80 x1875062187350784/t77309411525(77309411525) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201448.629897:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.629915:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.629971:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1251 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.629997:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1251 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.630008:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1251 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.630028:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1b100 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187351168:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.630102:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63e40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.633550:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f1b100 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187351168:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.633588:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1251 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8b7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.633601:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1251 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8b7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.633613:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1251 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd8b7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.633624:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1251 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8b7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.633637:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1251 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd8b7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.633645:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1251 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd8b7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.633681:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1251 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd8b7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.633693:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290f1b100 x1875062187351168/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.633707:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1251 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8b7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.633720:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201448.633726:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201448.633855:0:41043:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201448.633861:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e1251 00000080:00010000:1.0:1788201448.633867:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.633870:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1251 from it ffffaa1800c97bf8 00010000:00010000:1.0:1788201448.633874:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1251 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd8b7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.633884:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1251 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8b7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.633905:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201448.633926:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.633940:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.633949:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1251 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8b7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.633970:0:41043:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1251 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8b7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.633977:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201448.633981:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.633983:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e1251 00000080:00010000:1.0:1788201448.633987:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1251 from it ffffaa1800c97b38 00010000:00010000:1.0:1788201448.633990:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1251 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd8b7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.634009:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1251 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8b7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201448.634018:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201448.634022:0:41043:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-2-49, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), file ffff8e82829b9b00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201448.634030:0:41043:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-2-49, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000002:00010000:1.0:1788201448.634047:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-2-49,[0x2000088d1:0x32:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201448.634057:0:41043:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1251 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbd8b7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.634102:0:41043:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1251 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbd8b7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.634109:0:41043:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201448.634112:0:41043:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1251 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbd8b7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.634155:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6c00/0x9ae3109cee9e1258 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.634162:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6c00/0x9ae3109cee9e1258 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201448.634172:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201448.634178:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290f1b100 x1875062187351424/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201448.634187:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6c00/0x9ae3109cee9e1258 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.634206:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1b100 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187351424:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.634356:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63e40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.638723:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f1b100 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187351424:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.638767:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6c00/0x9ae3109cee9e1258 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8d3 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.638794:0:41043:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6c00/0x9ae3109cee9e1258 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x32:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8d3 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.638803:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6c00/0x9ae3109cee9e1258 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8d3 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.638813:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6c00/0x9ae3109cee9e1258 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd8d3 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.638843:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6c00/0x9ae3109cee9e1258 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8d3 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.638857:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6c00/0x9ae3109cee9e1258 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd8d3 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.638865:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6c00/0x9ae3109cee9e1258 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd8d3 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.638872:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6c00/0x9ae3109cee9e1258 lrc: 4/1,0 mode: PR/PR res: [0x2000088d1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd8d3 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.638882:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8290f1b100 x1875062187351424/t77309411527(77309411527) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201448.638895:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290f1b100 x1875062187351424/t77309411527(77309411527) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.638907:0:41043:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6c00/0x9ae3109cee9e1258 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8d3 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201448.638920:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6c00/0x9ae3109cee9e1258 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8d3 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.638931:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589969617/50 (0 0) 00000080:00200000:1.0:1788201448.638937:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201448.638948:0:41043:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-2-49, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000080:00200000:1.0:1788201448.638980:0:41043:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000088d1:0x32:0x0](ffff8e82927f1988) 00000080:00200000:1.0:1788201448.639065:0:41043:0:(file.c:6843:ll_layout_conf()) [0x2000088d1:0x32:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201448.639070:0:41043:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2000088d1:0x32:0x0](ffff8e82927f1988) 00000080:00010000:1.0:1788201448.639073:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000088d1:0x32:0x0](ffff8e82927f1988) for lock 0x9ae3109cee9e1258 00000080:00010000:1.0:1788201448.639082:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000088d1:0x32:0x0](ffff8e82927f1988) 00000080:00200008:1.0:1788201448.639095:0:41043:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-2-49:[0x2000088d1:0x32:0x0](ffff8e82927f1988), flags 100102 00000002:00100000:1.0:1788201448.639116:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290f1b100 x1875062187351424/t77309411527(77309411527) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201448.639146:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1258 from it ffff8e8287377240 00010000:00010000:1.0:1788201448.639151:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6c00/0x9ae3109cee9e1258 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbd8d3 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.639159:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6c00/0x9ae3109cee9e1258 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbd8d3 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.639180:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000088d1:0x32:0x0](ffff8e82927f1988), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201448.639211:0:41043:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-2-49:[0x2000088d1:0x32:0x0](ffff8e82927f1988), flags 100002 00000002:00100000:1.0:1788201448.639254:0:41043:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8290f1b100 x1875062187351424/t77309411527(77309411527) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201448.639263:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201448.639267:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290f19500 x1875062187351680/t0(0) o35->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201448.639285:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f19500 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187351680:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201448.639374:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a630c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.642218:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f19500 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187351680:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201448.642268:0:41043:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8290f1b100 x1875062187351424/t77309411527(77309411527) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201448.642384:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.642402:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.642457:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e125f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.642467:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e125f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.642476:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e125f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.642496:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1bb80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187351808:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.642555:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a630c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.646494:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f1bb80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187351808:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.646522:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e125f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8e8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.646534:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e125f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8e8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.646545:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e125f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd8e8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.646556:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e125f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8e8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.646719:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e125f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd8e8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.646729:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e125f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd8e8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.646737:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e125f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd8e8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.646825:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290f1bb80 x1875062187351808/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.646839:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e125f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8e8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.646851:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201448.646858:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201448.646884:0:41043:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201448.646890:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e125f 00000080:00010000:1.0:1788201448.646895:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.646899:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e125f from it ffffaa1800c97bf8 00010000:00010000:1.0:1788201448.646902:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e125f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd8e8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.646911:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e125f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8e8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.646946:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201448.646967:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.646981:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.646990:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e125f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8e8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.647085:0:41043:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e125f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8e8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.647093:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201448.647098:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.647101:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e125f 00000080:00010000:1.0:1788201448.647105:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e125f from it ffffaa1800c97b38 00010000:00010000:1.0:1788201448.647108:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e125f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd8e8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.647116:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e125f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd8e8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201448.647125:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201448.647130:0:41043:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-2-50, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), file ffff8e82829b9380, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201448.647139:0:41043:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-2-50, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000002:00010000:1.0:1788201448.647158:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-2-50,[0x2000088d1:0x33:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201448.647170:0:41043:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e125f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbd8e8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.647234:0:41043:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e125f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbd8e8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.647242:0:41043:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201448.647244:0:41043:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e125f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbd8e8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.647277:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6600/0x9ae3109cee9e1266 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.647284:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6600/0x9ae3109cee9e1266 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201448.647296:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201448.647303:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290f1bb80 x1875062187352064/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201448.647338:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6600/0x9ae3109cee9e1266 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.647359:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1bb80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187352064:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.647535:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a630c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.659931:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f1bb80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187352064:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.659960:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6600/0x9ae3109cee9e1266 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd904 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.659991:0:41043:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6600/0x9ae3109cee9e1266 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x33:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd904 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.660191:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6600/0x9ae3109cee9e1266 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd904 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.660202:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6600/0x9ae3109cee9e1266 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd904 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.660212:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6600/0x9ae3109cee9e1266 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd904 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.660227:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6600/0x9ae3109cee9e1266 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd904 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.660236:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6600/0x9ae3109cee9e1266 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd904 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.660243:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6600/0x9ae3109cee9e1266 lrc: 4/1,0 mode: PR/PR res: [0x2000088d1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbd904 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.660254:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8290f1bb80 x1875062187352064/t77309411529(77309411529) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201448.660267:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290f1bb80 x1875062187352064/t77309411529(77309411529) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.660281:0:41043:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6600/0x9ae3109cee9e1266 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd904 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201448.660294:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6600/0x9ae3109cee9e1266 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbd904 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.664522:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589969617/51 (0 0) 00000080:00200000:1.0:1788201448.664542:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201448.664557:0:41043:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-2-50, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000100:00100000:1.0:1788201448.882641:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b90e00 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187363200:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201448.882682:0:41043:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8288bae300 x1875062187362944/t77309411563(77309411563) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201448.882766:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.882784:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.882845:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e135b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.882854:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e135b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.882863:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e135b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.882880:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1aa00 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187363328:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.882986:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a630c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.885493:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f1aa00 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187363328:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.885517:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e135b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc5a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.885528:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e135b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc5a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.885555:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e135b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdc5a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.885565:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e135b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc5a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.885640:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e135b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdc5a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.885705:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e135b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdc5a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.885715:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e135b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdc5a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.885727:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290f1aa00 x1875062187363328/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.885739:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e135b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc5a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.885761:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201448.885768:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201448.885794:0:41043:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201448.885800:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e135b 00000080:00010000:1.0:1788201448.885805:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.885808:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e135b from it ffffaa1800c97bf8 00010000:00010000:1.0:1788201448.885817:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e135b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdc5a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.885826:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e135b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc5a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.885848:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201448.885871:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.885883:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.885892:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e135b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc5a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.885901:0:41043:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e135b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc5a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.885908:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201448.885911:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.885914:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e135b 00000080:00010000:1.0:1788201448.885917:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e135b from it ffffaa1800c97b38 00010000:00010000:1.0:1788201448.885921:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e135b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdc5a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.885927:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e135b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc5a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201448.885935:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201448.885940:0:41043:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-2-68, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), file ffff8e82829b9b00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201448.885947:0:41043:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-2-68, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000002:00010000:1.0:1788201448.885972:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-2-68,[0x2000088d1:0x45:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201448.885984:0:41043:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e135b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbdc5a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.886138:0:41043:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e135b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbdc5a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.886146:0:41043:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201448.886149:0:41043:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e135b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbdc5a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.886181:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7c00/0x9ae3109cee9e1362 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.886187:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7c00/0x9ae3109cee9e1362 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201448.886198:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201448.886205:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290f1aa00 x1875062187363584/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201448.886214:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7c00/0x9ae3109cee9e1362 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.886233:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1aa00 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187363584:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.886531:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a630c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.890232:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f1aa00 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187363584:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.890259:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7c00/0x9ae3109cee9e1362 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc76 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.890302:0:41043:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7c00/0x9ae3109cee9e1362 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x45:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc76 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.890436:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7c00/0x9ae3109cee9e1362 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc76 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.890447:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7c00/0x9ae3109cee9e1362 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdc76 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.890458:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7c00/0x9ae3109cee9e1362 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc76 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.890472:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7c00/0x9ae3109cee9e1362 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdc76 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.890480:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7c00/0x9ae3109cee9e1362 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdc76 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.890488:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7c00/0x9ae3109cee9e1362 lrc: 4/1,0 mode: PR/PR res: [0x2000088d1:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdc76 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.890500:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8290f1aa00 x1875062187363584/t77309411565(77309411565) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201448.890513:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290f1aa00 x1875062187363584/t77309411565(77309411565) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.890526:0:41043:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7c00/0x9ae3109cee9e1362 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc76 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201448.890546:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7c00/0x9ae3109cee9e1362 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc76 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.890557:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589969617/69 (0 0) 00000080:00200000:1.0:1788201448.890563:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201448.890635:0:41043:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-2-68, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000080:00200000:1.0:1788201448.890725:0:41043:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000088d1:0x45:0x0](ffff8e82927f4b08) 00000080:00200000:1.0:1788201448.890808:0:41043:0:(file.c:6843:ll_layout_conf()) [0x2000088d1:0x45:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201448.890814:0:41043:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2000088d1:0x45:0x0](ffff8e82927f4b08) 00000080:00010000:1.0:1788201448.890817:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000088d1:0x45:0x0](ffff8e82927f4b08) for lock 0x9ae3109cee9e1362 00000080:00010000:1.0:1788201448.890826:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000088d1:0x45:0x0](ffff8e82927f4b08) 00000080:00200008:1.0:1788201448.890840:0:41043:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-2-68:[0x2000088d1:0x45:0x0](ffff8e82927f4b08), flags 100102 00000002:00100000:1.0:1788201448.890862:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290f1aa00 x1875062187363584/t77309411565(77309411565) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201448.890891:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1362 from it ffff8e8287377000 00010000:00010000:1.0:1788201448.890895:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7c00/0x9ae3109cee9e1362 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbdc76 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.890904:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7c00/0x9ae3109cee9e1362 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x45:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbdc76 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.890923:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000088d1:0x45:0x0](ffff8e82927f4b08), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201448.890954:0:41043:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-2-68:[0x2000088d1:0x45:0x0](ffff8e82927f4b08), flags 100002 00000002:00100000:1.0:1788201448.891097:0:41043:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8290f1aa00 x1875062187363584/t77309411565(77309411565) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201448.891121:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201448.891127:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290f18a80 x1875062187363840/t0(0) o35->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201448.891147:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f18a80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187363840:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201448.891274:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63780 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.893826:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f18a80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187363840:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201448.893859:0:41043:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8290f1aa00 x1875062187363584/t77309411565(77309411565) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201448.893935:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.893952:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.894192:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e1369 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.894201:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e1369 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.894211:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e1369 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.894230:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1a300 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187363968:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.894487:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63780 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.897408:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f1a300 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187363968:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.897431:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e1369 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc8b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.897461:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e1369 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc8b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.897472:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e1369 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdc8b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.897482:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e1369 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc8b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.897494:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e1369 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdc8b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.897502:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e1369 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdc8b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.897508:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e1369 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdc8b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.897519:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290f1a300 x1875062187363968/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.897532:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e1369 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc8b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.897543:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201448.897550:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201448.897574:0:41043:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201448.897596:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e1369 00000080:00010000:1.0:1788201448.897602:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.897605:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1369 from it ffffaa1800c97bf8 00010000:00010000:1.0:1788201448.897608:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e1369 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdc8b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.897616:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e1369 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc8b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.897643:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201448.897674:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.897688:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.897698:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e1369 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc8b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.897707:0:41043:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e1369 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc8b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.897714:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201448.897719:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.897722:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e1369 00000080:00010000:1.0:1788201448.897725:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1369 from it ffffaa1800c97b38 00010000:00010000:1.0:1788201448.897728:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e1369 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdc8b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.897734:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e1369 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdc8b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201448.897763:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201448.897769:0:41043:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-2-69, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), file ffff8e82829b9980, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201448.897777:0:41043:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-2-69, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000002:00010000:1.0:1788201448.897796:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-2-69,[0x2000088d1:0x46:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201448.897807:0:41043:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e1369 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbdc8b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.897858:0:41043:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e1369 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbdc8b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.897865:0:41043:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201448.897868:0:41043:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e1369 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbdc8b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.897898:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e1370 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.897904:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e1370 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201448.897914:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201448.897921:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290f1a300 x1875062187364224/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201448.897929:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e1370 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.897947:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1a300 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187364224:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.898076:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63780 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.902048:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f1a300 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187364224:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.902079:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e1370 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdca7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.902105:0:41043:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e1370 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x46:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdca7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.902114:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e1370 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdca7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.902124:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e1370 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdca7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.902133:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e1370 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdca7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.902146:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e1370 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdca7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.902155:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e1370 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdca7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.902161:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e1370 lrc: 4/1,0 mode: PR/PR res: [0x2000088d1:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdca7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.902172:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8290f1a300 x1875062187364224/t77309411567(77309411567) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201448.902199:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290f1a300 x1875062187364224/t77309411567(77309411567) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.902214:0:41043:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e1370 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdca7 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201448.902225:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e1370 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdca7 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.902236:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589969617/70 (0 0) 00000080:00200000:1.0:1788201448.902242:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201448.902255:0:41043:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-2-69, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000080:00200000:1.0:1788201448.902286:0:41043:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000088d1:0x46:0x0](ffff8e82927f00c8) 00000080:00200000:1.0:1788201448.902388:0:41043:0:(file.c:6843:ll_layout_conf()) [0x2000088d1:0x46:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201448.902394:0:41043:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2000088d1:0x46:0x0](ffff8e82927f00c8) 00000080:00010000:1.0:1788201448.902398:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000088d1:0x46:0x0](ffff8e82927f00c8) for lock 0x9ae3109cee9e1370 00000080:00010000:1.0:1788201448.902408:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000088d1:0x46:0x0](ffff8e82927f00c8) 00000080:00200008:1.0:1788201448.902421:0:41043:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-2-69:[0x2000088d1:0x46:0x0](ffff8e82927f00c8), flags 100102 00000002:00100000:1.0:1788201448.902441:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290f1a300 x1875062187364224/t77309411567(77309411567) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201448.902472:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1370 from it ffff8e8287377000 00010000:00010000:1.0:1788201448.902477:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e1370 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbdca7 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.902486:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7200/0x9ae3109cee9e1370 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x46:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbdca7 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.902515:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000088d1:0x46:0x0](ffff8e82927f00c8), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201448.902542:0:41043:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-2-69:[0x2000088d1:0x46:0x0](ffff8e82927f00c8), flags 100002 00000002:00100000:1.0:1788201448.902595:0:41043:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8290f1a300 x1875062187364224/t77309411567(77309411567) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201448.902605:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201448.902610:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290f18700 x1875062187364480/t0(0) o35->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201448.902629:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f18700 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187364480:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201448.902692:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63540 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.905057:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f18700 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187364480:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201448.905086:0:41043:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8290f1a300 x1875062187364224/t77309411567(77309411567) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201448.905152:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.905169:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.905219:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1377 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.905228:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1377 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.905236:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1377 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.905254:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f19c00 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187364608:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.905349:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63540 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.908177:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f19c00 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187364608:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.908201:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1377 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdcbc expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.908214:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1377 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdcbc expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.908225:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1377 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdcbc expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.908236:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1377 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdcbc expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.908248:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1377 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdcbc expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.908255:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1377 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdcbc expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.908264:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1377 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdcbc expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.908275:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290f19c00 x1875062187364608/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.908288:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1377 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdcbc expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.908362:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201448.908368:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201448.908392:0:41043:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201448.908398:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e1377 00000080:00010000:1.0:1788201448.908404:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.908407:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1377 from it ffffaa1800c97bf8 00010000:00010000:1.0:1788201448.908411:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1377 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdcbc expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.908419:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1377 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdcbc expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.908442:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201448.908465:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.908478:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.908487:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1377 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdcbc expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.908498:0:41043:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1377 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdcbc expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.908506:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201448.908510:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.908513:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e1377 00000080:00010000:1.0:1788201448.908516:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1377 from it ffffaa1800c97b38 00010000:00010000:1.0:1788201448.908520:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1377 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdcbc expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.908534:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1377 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdcbc expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201448.908543:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201448.908548:0:41043:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-2-70, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), file ffff8e82829b9b00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201448.908556:0:41043:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-2-70, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000002:00010000:1.0:1788201448.908574:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-2-70,[0x2000088d1:0x47:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201448.908597:0:41043:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1377 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbdcbc expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.908658:0:41043:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1377 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbdcbc expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.908666:0:41043:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201448.908669:0:41043:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1377 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbdcbc expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.908699:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e137e lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.908707:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e137e lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201448.908717:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201448.908724:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290f19c00 x1875062187364864/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201448.908733:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e137e lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.908768:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f19c00 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187364864:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.908880:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63540 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.917512:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f19c00 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187364864:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.917538:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e137e lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdcd8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.917565:0:41043:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e137e lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x47:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdcd8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.917574:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e137e lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdcd8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.917598:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e137e lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdcd8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.917610:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e137e lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdcd8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.917623:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e137e lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdcd8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.917631:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e137e lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdcd8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.917639:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e137e lrc: 4/1,0 mode: PR/PR res: [0x2000088d1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdcd8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.917679:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8290f19c00 x1875062187364864/t77309411569(77309411569) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201448.917692:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290f19c00 x1875062187364864/t77309411569(77309411569) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.917706:0:41043:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e137e lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdcd8 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201448.917719:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e137e lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdcd8 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.917730:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589969617/71 (0 0) 00000080:00200000:1.0:1788201448.917736:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201448.917760:0:41043:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-2-70, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000080:00200000:1.0:1788201448.917792:0:41043:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000088d1:0x47:0x0](ffff8e82927f2a08) 00000080:00200000:1.0:1788201448.917864:0:41043:0:(file.c:6843:ll_layout_conf()) [0x2000088d1:0x47:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201448.917868:0:41043:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2000088d1:0x47:0x0](ffff8e82927f2a08) 00000080:00010000:1.0:1788201448.917871:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000088d1:0x47:0x0](ffff8e82927f2a08) for lock 0x9ae3109cee9e137e 00000080:00010000:1.0:1788201448.917880:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000088d1:0x47:0x0](ffff8e82927f2a08) 00000080:00200008:1.0:1788201448.917892:0:41043:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-2-70:[0x2000088d1:0x47:0x0](ffff8e82927f2a08), flags 100102 00000002:00100000:1.0:1788201448.917912:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290f19c00 x1875062187364864/t77309411569(77309411569) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201448.917941:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e137e from it ffff8e8287377000 00010000:00010000:1.0:1788201448.917946:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e137e lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbdcd8 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.917965:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7800/0x9ae3109cee9e137e lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x47:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbdcd8 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.917984:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000088d1:0x47:0x0](ffff8e82927f2a08), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201448.918026:0:41043:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-2-70:[0x2000088d1:0x47:0x0](ffff8e82927f2a08), flags 100002 00000002:00100000:1.0:1788201448.918065:0:41043:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8290f19c00 x1875062187364864/t77309411569(77309411569) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201448.918076:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201448.918081:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290f1b800 x1875062187365120/t0(0) o35->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201448.918100:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1b800 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187365120:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201448.918180:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.921148:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f1b800 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187365120:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201448.921181:0:41043:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8290f19c00 x1875062187364864/t77309411569(77309411569) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201448.921256:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.921273:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.921356:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1385 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.921365:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1385 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.921373:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1385 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.921406:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f18e00 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187365248:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.921479:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.924559:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f18e00 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187365248:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.924597:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1385 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdced expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.924609:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1385 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdced expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.924619:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1385 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdced expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.924629:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1385 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdced expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.924641:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1385 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdced expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.924658:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1385 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdced expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.924665:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1385 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdced expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.924675:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290f18e00 x1875062187365248/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.924703:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1385 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdced expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.924715:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201448.924720:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201448.924744:0:41043:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201448.924762:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e1385 00000080:00010000:1.0:1788201448.924767:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.924770:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1385 from it ffffaa1800c97bf8 00010000:00010000:1.0:1788201448.924773:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1385 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdced expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.924781:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1385 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdced expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.924802:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201448.924824:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.924837:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.924846:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1385 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdced expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.924856:0:41043:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1385 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdced expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.924862:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201448.924866:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.924869:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e1385 00000080:00010000:1.0:1788201448.924879:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1385 from it ffffaa1800c97b38 00010000:00010000:1.0:1788201448.924882:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1385 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdced expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.924890:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1385 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdced expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201448.924899:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201448.924904:0:41043:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-2-71, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), file ffff8e82829b9980, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201448.924911:0:41043:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-2-71, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000002:00010000:1.0:1788201448.924930:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-2-71,[0x2000088d1:0x48:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201448.924941:0:41043:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1385 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbdced expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.924993:0:41043:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1385 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbdced expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.925015:0:41043:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201448.925018:0:41043:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1385 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbdced expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.925052:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6400/0x9ae3109cee9e138c lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.925060:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6400/0x9ae3109cee9e138c lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201448.925070:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201448.925076:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290f18e00 x1875062187365504/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201448.925093:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6400/0x9ae3109cee9e138c lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.925114:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f18e00 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187365504:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.925235:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.929633:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f18e00 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187365504:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.929671:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6400/0x9ae3109cee9e138c lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd09 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.929700:0:41043:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6400/0x9ae3109cee9e138c lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x48:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd09 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.929709:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6400/0x9ae3109cee9e138c lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd09 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.929721:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6400/0x9ae3109cee9e138c lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd09 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.929731:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6400/0x9ae3109cee9e138c lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd09 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.929745:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6400/0x9ae3109cee9e138c lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd09 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.929764:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6400/0x9ae3109cee9e138c lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd09 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.929792:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6400/0x9ae3109cee9e138c lrc: 4/1,0 mode: PR/PR res: [0x2000088d1:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd09 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.929806:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8290f18e00 x1875062187365504/t77309411571(77309411571) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201448.929819:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290f18e00 x1875062187365504/t77309411571(77309411571) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.929835:0:41043:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6400/0x9ae3109cee9e138c lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd09 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201448.929848:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6400/0x9ae3109cee9e138c lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd09 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.929858:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589969617/72 (0 0) 00000080:00200000:1.0:1788201448.929864:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201448.929876:0:41043:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-2-71, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000080:00200000:1.0:1788201448.929903:0:41043:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000088d1:0x48:0x0](ffff8e82927f3248) 00000080:00200000:1.0:1788201448.929973:0:41043:0:(file.c:6843:ll_layout_conf()) [0x2000088d1:0x48:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201448.929978:0:41043:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2000088d1:0x48:0x0](ffff8e82927f3248) 00000080:00010000:1.0:1788201448.929981:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000088d1:0x48:0x0](ffff8e82927f3248) for lock 0x9ae3109cee9e138c 00000080:00010000:1.0:1788201448.929991:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000088d1:0x48:0x0](ffff8e82927f3248) 00000080:00200008:1.0:1788201448.930019:0:41043:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-2-71:[0x2000088d1:0x48:0x0](ffff8e82927f3248), flags 100102 00000002:00100000:1.0:1788201448.930041:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290f18e00 x1875062187365504/t77309411571(77309411571) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201448.930084:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e138c from it ffff8e8287377360 00010000:00010000:1.0:1788201448.930089:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6400/0x9ae3109cee9e138c lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbdd09 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.930100:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6400/0x9ae3109cee9e138c lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x48:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbdd09 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.930121:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000088d1:0x48:0x0](ffff8e82927f3248), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201448.930150:0:41043:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-2-71:[0x2000088d1:0x48:0x0](ffff8e82927f3248), flags 100002 00000002:00100000:1.0:1788201448.930192:0:41043:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8290f18e00 x1875062187365504/t77309411571(77309411571) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201448.930203:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201448.930208:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290f1a680 x1875062187365760/t0(0) o35->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201448.930230:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1a680 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187365760:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201448.930301:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63240 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.933563:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f1a680 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187365760:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201448.933605:0:41043:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8290f18e00 x1875062187365504/t77309411571(77309411571) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201448.933685:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.933703:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.933768:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1393 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.933794:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1393 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.933804:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1393 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.933823:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f19180 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187365888:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.933895:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63240 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.936388:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f19180 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187365888:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.936412:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1393 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd1e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.936423:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1393 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd1e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.936435:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1393 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd1e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.936446:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1393 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd1e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.936458:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1393 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd1e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.936466:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1393 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd1e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.936492:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1393 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd1e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.936504:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290f19180 x1875062187365888/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.936517:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1393 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd1e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.936528:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201448.936534:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201448.936557:0:41043:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201448.936563:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e1393 00000080:00010000:1.0:1788201448.936568:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.936571:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1393 from it ffffaa1800c97bf8 00010000:00010000:1.0:1788201448.936585:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1393 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd1e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.936594:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1393 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd1e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.936618:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201448.936640:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.936653:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.936670:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1393 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd1e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.936693:0:41043:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1393 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd1e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.936700:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201448.936705:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.936708:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e1393 00000080:00010000:1.0:1788201448.936712:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1393 from it ffffaa1800c97b38 00010000:00010000:1.0:1788201448.936714:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1393 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd1e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.936722:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1393 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd1e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201448.936730:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201448.936736:0:41043:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-2-72, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), file ffff8e82829b9b00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201448.936743:0:41043:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-2-72, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000002:00010000:1.0:1788201448.936772:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-2-72,[0x2000088d1:0x49:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201448.936783:0:41043:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1393 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbdd1e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.936835:0:41043:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1393 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbdd1e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.936841:0:41043:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201448.936844:0:41043:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e1393 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbdd1e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.936887:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6e00/0x9ae3109cee9e139a lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.936894:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6e00/0x9ae3109cee9e139a lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201448.936905:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201448.936911:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290f19180 x1875062187366144/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201448.936920:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6e00/0x9ae3109cee9e139a lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.936938:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f19180 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187366144:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.937053:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63240 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.940928:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f19180 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187366144:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.940956:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6e00/0x9ae3109cee9e139a lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd3a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.940982:0:41043:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6e00/0x9ae3109cee9e139a lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x49:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd3a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.940990:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6e00/0x9ae3109cee9e139a lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd3a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.941015:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6e00/0x9ae3109cee9e139a lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd3a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.941043:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6e00/0x9ae3109cee9e139a lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd3a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.941058:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6e00/0x9ae3109cee9e139a lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd3a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.941066:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6e00/0x9ae3109cee9e139a lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd3a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.941074:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6e00/0x9ae3109cee9e139a lrc: 4/1,0 mode: PR/PR res: [0x2000088d1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd3a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.941086:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8290f19180 x1875062187366144/t77309411573(77309411573) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201448.941098:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290f19180 x1875062187366144/t77309411573(77309411573) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.941110:0:41043:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6e00/0x9ae3109cee9e139a lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd3a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201448.941123:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6e00/0x9ae3109cee9e139a lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd3a expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.941133:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589969617/73 (0 0) 00000080:00200000:1.0:1788201448.941139:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201448.941150:0:41043:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-2-72, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000080:00200000:1.0:1788201448.941186:0:41043:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000088d1:0x49:0x0](ffff8e82927f1148) 00000080:00200000:1.0:1788201448.941255:0:41043:0:(file.c:6843:ll_layout_conf()) [0x2000088d1:0x49:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201448.941260:0:41043:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2000088d1:0x49:0x0](ffff8e82927f1148) 00000080:00010000:1.0:1788201448.941263:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000088d1:0x49:0x0](ffff8e82927f1148) for lock 0x9ae3109cee9e139a 00000080:00010000:1.0:1788201448.941273:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000088d1:0x49:0x0](ffff8e82927f1148) 00000080:00200008:1.0:1788201448.941286:0:41043:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-2-72:[0x2000088d1:0x49:0x0](ffff8e82927f1148), flags 100102 00000002:00100000:1.0:1788201448.941346:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290f19180 x1875062187366144/t77309411573(77309411573) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201448.941377:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e139a from it ffff8e8287377360 00010000:00010000:1.0:1788201448.941382:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6e00/0x9ae3109cee9e139a lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbdd3a expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.941391:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6e00/0x9ae3109cee9e139a lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x49:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbdd3a expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.941410:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000088d1:0x49:0x0](ffff8e82927f1148), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201448.941439:0:41043:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-2-72:[0x2000088d1:0x49:0x0](ffff8e82927f1148), flags 100002 00000002:00100000:1.0:1788201448.941484:0:41043:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8290f19180 x1875062187366144/t77309411573(77309411573) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201448.941494:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201448.941499:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290f1b480 x1875062187366400/t0(0) o35->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201448.941520:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290f1b480 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187366400:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201448.941602:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63600 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.944516:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f1b480 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187366400:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201448.944570:0:41043:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8290f19180 x1875062187366144/t77309411573(77309411573) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201448.944670:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.944688:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.944743:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13a1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.944764:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13a1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.944775:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13a1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.944792:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe2300 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187366528:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.944856:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63600 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.947902:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe2300 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187366528:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.947930:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13a1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd4f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.947942:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13a1 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd4f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.947954:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13a1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd4f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.947964:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13a1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd4f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.947996:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13a1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd4f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.948004:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13a1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd4f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.948011:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13a1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd4f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.948022:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290fe2300 x1875062187366528/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.948035:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13a1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd4f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.948045:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201448.948051:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201448.948077:0:41043:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201448.948084:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e13a1 00000080:00010000:1.0:1788201448.948088:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.948091:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e13a1 from it ffffaa1800c97bf8 00010000:00010000:1.0:1788201448.948094:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13a1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd4f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.948102:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13a1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd4f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.948134:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201448.948154:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.948169:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.948178:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13a1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd4f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.948188:0:41043:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13a1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd4f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.948194:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201448.948199:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.948201:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e13a1 00000080:00010000:1.0:1788201448.948205:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e13a1 from it ffffaa1800c97b38 00010000:00010000:1.0:1788201448.948208:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13a1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd4f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.948216:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13a1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd4f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201448.948224:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201448.948230:0:41043:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-2-73, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), file ffff8e82829b9980, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201448.948238:0:41043:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-2-73, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000002:00010000:1.0:1788201448.948258:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-2-73,[0x2000088d1:0x4a:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201448.948268:0:41043:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13a1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbdd4f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.948369:0:41043:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13a1 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbdd4f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.948377:0:41043:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201448.948380:0:41043:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13a1 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbdd4f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.948414:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e13a8 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.948421:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e13a8 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201448.948431:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201448.948437:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe2300 x1875062187366784/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201448.948446:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e13a8 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.948463:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe2300 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187366784:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.948591:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63600 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.952245:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe2300 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187366784:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.952271:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e13a8 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd6b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.952300:0:41043:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e13a8 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x4a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd6b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.952355:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e13a8 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd6b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.952367:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e13a8 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd6b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.952378:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e13a8 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd6b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.952392:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e13a8 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd6b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.952401:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e13a8 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd6b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.952409:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e13a8 lrc: 4/1,0 mode: PR/PR res: [0x2000088d1:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd6b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.952421:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8290fe2300 x1875062187366784/t77309411575(77309411575) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201448.952435:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290fe2300 x1875062187366784/t77309411575(77309411575) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.952447:0:41043:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e13a8 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd6b expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201448.952461:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e13a8 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd6b expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.952478:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589969617/74 (0 0) 00000080:00200000:1.0:1788201448.952485:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201448.952498:0:41043:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-2-73, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000080:00200000:1.0:1788201448.952528:0:41043:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000088d1:0x4a:0x0](ffff8e82927f6c08) 00000080:00200000:1.0:1788201448.952611:0:41043:0:(file.c:6843:ll_layout_conf()) [0x2000088d1:0x4a:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201448.952616:0:41043:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2000088d1:0x4a:0x0](ffff8e82927f6c08) 00000080:00010000:1.0:1788201448.952620:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000088d1:0x4a:0x0](ffff8e82927f6c08) for lock 0x9ae3109cee9e13a8 00000080:00010000:1.0:1788201448.952629:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000088d1:0x4a:0x0](ffff8e82927f6c08) 00000080:00200008:1.0:1788201448.952643:0:41043:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-2-73:[0x2000088d1:0x4a:0x0](ffff8e82927f6c08), flags 100102 00000002:00100000:1.0:1788201448.952675:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290fe2300 x1875062187366784/t77309411575(77309411575) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201448.952707:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e13a8 from it ffff8e8287377360 00010000:00010000:1.0:1788201448.952712:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e13a8 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbdd6b expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.952722:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e13a8 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbdd6b expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.952743:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000088d1:0x4a:0x0](ffff8e82927f6c08), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201448.952785:0:41043:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-2-73:[0x2000088d1:0x4a:0x0](ffff8e82927f6c08), flags 100002 00000002:00100000:1.0:1788201448.952826:0:41043:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8290fe2300 x1875062187366784/t77309411575(77309411575) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201448.952837:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201448.952842:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe1c00 x1875062187367040/t0(0) o35->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201448.952874:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe1c00 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187367040:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201448.952944:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63f00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.955194:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe1c00 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187367040:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201448.955227:0:41043:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8290fe2300 x1875062187366784/t77309411575(77309411575) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201448.955301:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.955349:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.955406:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13af lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.955416:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13af lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.955427:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13af lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.955445:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe0700 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187367168:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.955513:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63f00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.958835:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe0700 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187367168:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.958864:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13af lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd80 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.958876:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13af lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd80 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.958908:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13af lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd80 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.958920:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13af lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd80 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.958934:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13af lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd80 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.958941:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13af lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd80 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.958949:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13af lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd80 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.958961:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290fe0700 x1875062187367168/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.958975:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13af lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd80 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.958988:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201448.958994:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201448.959019:0:41043:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201448.959024:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e13af 00000080:00010000:1.0:1788201448.959029:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.959032:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e13af from it ffffaa1800c97bf8 00010000:00010000:1.0:1788201448.959041:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13af lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd80 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.959051:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13af lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd80 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.959074:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201448.959097:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.959112:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.959121:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd80 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.959131:0:41043:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd80 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.959137:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201448.959141:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.959144:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e13af 00000080:00010000:1.0:1788201448.959147:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e13af from it ffffaa1800c97b38 00010000:00010000:1.0:1788201448.959150:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13af lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd80 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.959157:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13af lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd80 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201448.959165:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201448.959170:0:41043:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-2-74, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), file ffff8e82829b9b00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201448.959178:0:41043:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-2-74, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000002:00010000:1.0:1788201448.959203:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-2-74,[0x2000088d1:0x4b:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201448.959215:0:41043:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13af lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbdd80 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.959268:0:41043:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13af lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbdd80 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.959275:0:41043:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201448.959277:0:41043:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13af lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbdd80 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.959329:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13b6 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.959336:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13b6 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201448.959346:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201448.959352:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe0700 x1875062187367424/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201448.959360:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13b6 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.959378:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe0700 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187367424:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.959496:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63f00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.963544:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe0700 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187367424:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.963571:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13b6 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd9c expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.963629:0:41043:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13b6 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x4b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd9c expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.963637:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13b6 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd9c expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.963647:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13b6 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd9c expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.963667:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13b6 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd9c expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.963682:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13b6 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd9c expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.963689:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13b6 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd9c expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.963696:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13b6 lrc: 4/1,0 mode: PR/PR res: [0x2000088d1:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdd9c expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.963706:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8290fe0700 x1875062187367424/t77309411577(77309411577) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201448.963718:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290fe0700 x1875062187367424/t77309411577(77309411577) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.963730:0:41043:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13b6 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd9c expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201448.963759:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13b6 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdd9c expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.963771:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589969617/75 (0 0) 00000080:00200000:1.0:1788201448.963777:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201448.963789:0:41043:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-2-74, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000080:00200000:1.0:1788201448.963815:0:41043:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000088d1:0x4b:0x0](ffff8e82927f0908) 00000080:00200000:1.0:1788201448.963883:0:41043:0:(file.c:6843:ll_layout_conf()) [0x2000088d1:0x4b:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201448.963889:0:41043:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2000088d1:0x4b:0x0](ffff8e82927f0908) 00000080:00010000:1.0:1788201448.963892:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000088d1:0x4b:0x0](ffff8e82927f0908) for lock 0x9ae3109cee9e13b6 00000080:00010000:1.0:1788201448.963900:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000088d1:0x4b:0x0](ffff8e82927f0908) 00000080:00200008:1.0:1788201448.963915:0:41043:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-2-74:[0x2000088d1:0x4b:0x0](ffff8e82927f0908), flags 100102 00000002:00100000:1.0:1788201448.963936:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290fe0700 x1875062187367424/t77309411577(77309411577) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201448.963964:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e13b6 from it ffff8e8287377360 00010000:00010000:1.0:1788201448.963968:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13b6 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbdd9c expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.963977:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13b6 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbdd9c expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.964226:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000088d1:0x4b:0x0](ffff8e82927f0908), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201448.964261:0:41043:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-2-74:[0x2000088d1:0x4b:0x0](ffff8e82927f0908), flags 100002 00000002:00100000:1.0:1788201448.964337:0:41043:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8290fe0700 x1875062187367424/t77309411577(77309411577) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201448.964360:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201448.964366:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe0a80 x1875062187367680/t0(0) o35->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201448.964387:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe0a80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187367680:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201448.964474:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63000 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.967348:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe0a80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187367680:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201448.967385:0:41043:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8290fe0700 x1875062187367424/t77309411577(77309411577) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201448.967459:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.967479:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.967533:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13bd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.967541:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13bd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.967551:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13bd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.967568:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe3b80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187367808:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.967642:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63000 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.971366:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe3b80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187367808:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.971395:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13bd lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbddb1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.971428:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13bd lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbddb1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.971441:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13bd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbddb1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.971452:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13bd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbddb1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.971465:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13bd lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbddb1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.971474:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13bd lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbddb1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.971481:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13bd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbddb1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.971493:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290fe3b80 x1875062187367808/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.971507:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13bd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbddb1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.971520:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201448.971526:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201448.971554:0:41043:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201448.971566:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e13bd 00000080:00010000:1.0:1788201448.971571:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.971574:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e13bd from it ffffaa1800c97bf8 00010000:00010000:1.0:1788201448.971589:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13bd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbddb1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.971599:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13bd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbddb1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.971625:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201448.971649:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.971664:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.971673:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13bd lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbddb1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.971683:0:41043:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13bd lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbddb1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.971689:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201448.971692:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.971695:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e13bd 00000080:00010000:1.0:1788201448.971698:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e13bd from it ffffaa1800c97b38 00010000:00010000:1.0:1788201448.971701:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13bd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbddb1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.971708:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13bd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbddb1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201448.971726:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201448.971731:0:41043:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-2-75, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), file ffff8e82829b9980, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201448.971740:0:41043:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-2-75, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000002:00010000:1.0:1788201448.971770:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-2-75,[0x2000088d1:0x4c:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201448.971782:0:41043:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13bd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbddb1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.971836:0:41043:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13bd lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbddb1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.971844:0:41043:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201448.971847:0:41043:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13bd lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbddb1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201448.971882:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7600/0x9ae3109cee9e13c4 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.971889:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7600/0x9ae3109cee9e13c4 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201448.971899:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201448.971906:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe3b80 x1875062187368064/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201448.971915:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7600/0x9ae3109cee9e13c4 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.971934:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe3b80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187368064:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.972078:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63000 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.978528:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe3b80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187368064:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.978555:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7600/0x9ae3109cee9e13c4 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbddcd expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.978592:0:41043:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7600/0x9ae3109cee9e13c4 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x4c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbddcd expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.978601:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7600/0x9ae3109cee9e13c4 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbddcd expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.978612:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7600/0x9ae3109cee9e13c4 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbddcd expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.978622:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7600/0x9ae3109cee9e13c4 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbddcd expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.978635:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7600/0x9ae3109cee9e13c4 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbddcd expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.978644:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7600/0x9ae3109cee9e13c4 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbddcd expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.978659:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7600/0x9ae3109cee9e13c4 lrc: 4/1,0 mode: PR/PR res: [0x2000088d1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbddcd expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.978671:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8290fe3b80 x1875062187368064/t77309411579(77309411579) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201448.978701:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290fe3b80 x1875062187368064/t77309411579(77309411579) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.978716:0:41043:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7600/0x9ae3109cee9e13c4 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbddcd expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201448.978728:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7600/0x9ae3109cee9e13c4 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbddcd expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.978739:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589969617/76 (0 0) 00000080:00200000:1.0:1788201448.978745:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201448.978769:0:41043:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-2-75, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000080:00200000:1.0:1788201448.978799:0:41043:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000088d1:0x4c:0x0](ffff8e82927f7448) 00000080:00200000:1.0:1788201448.978874:0:41043:0:(file.c:6843:ll_layout_conf()) [0x2000088d1:0x4c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201448.978879:0:41043:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2000088d1:0x4c:0x0](ffff8e82927f7448) 00000080:00010000:1.0:1788201448.978882:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000088d1:0x4c:0x0](ffff8e82927f7448) for lock 0x9ae3109cee9e13c4 00000080:00010000:1.0:1788201448.978891:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000088d1:0x4c:0x0](ffff8e82927f7448) 00000080:00200008:1.0:1788201448.978906:0:41043:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-2-75:[0x2000088d1:0x4c:0x0](ffff8e82927f7448), flags 100102 00000002:00100000:1.0:1788201448.978925:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290fe3b80 x1875062187368064/t77309411579(77309411579) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201448.978955:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e13c4 from it ffff8e8287377360 00010000:00010000:1.0:1788201448.978960:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7600/0x9ae3109cee9e13c4 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbddcd expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.978969:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7600/0x9ae3109cee9e13c4 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbddcd expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.979139:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000088d1:0x4c:0x0](ffff8e82927f7448), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201448.979170:0:41043:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-2-75:[0x2000088d1:0x4c:0x0](ffff8e82927f7448), flags 100002 00000002:00100000:1.0:1788201448.979211:0:41043:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8290fe3b80 x1875062187368064/t77309411579(77309411579) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201448.979221:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201448.979226:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe1500 x1875062187368320/t0(0) o35->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201448.979244:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe1500 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187368320:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201448.979381:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a639c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.982065:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe1500 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187368320:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201448.982100:0:41043:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8290fe3b80 x1875062187368064/t77309411579(77309411579) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201448.982179:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.982199:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.982257:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13cb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.982266:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13cb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.982275:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13cb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201448.982295:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe2680 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187368448:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201448.982418:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a639c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201448.985240:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe2680 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187368448:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201448.985268:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13cb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdde2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.985282:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13cb lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdde2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.985293:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13cb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdde2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.985304:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13cb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdde2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.985337:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13cb lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdde2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.985345:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13cb lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdde2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.985351:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13cb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdde2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201448.985363:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290fe2680 x1875062187368448/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201464 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201448.985376:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13cb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdde2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.985410:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201448.985418:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201448.985447:0:41043:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201448.985453:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e13cb 00000080:00010000:1.0:1788201448.985459:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.985462:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e13cb from it ffffaa1800c97bf8 00010000:00010000:1.0:1788201448.985466:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13cb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdde2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201448.985474:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13cb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdde2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.985499:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201448.985522:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201448.985537:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201448.985546:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13cb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdde2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.985556:0:41043:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13cb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdde2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201448.985562:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201448.985566:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201448.985569:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e13cb 00000080:00010000:1.0:1788201448.985572:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e13cb from it ffffaa1800c97b38 00010000:00010000:1.0:1788201448.985589:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13cb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdde2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201448.985603:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e13cb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdde2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201448.985612:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201448.985618:0:41043:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-2-76, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), file ffff8e82829b9b00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201448.985626:0:41043:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-2-76, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000100:00100000:1.0:1788201449.112162:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b91500 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187373440:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201449.112199:0:41043:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8289510a80 x1875062187373184/t77309411595(77309411595) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201465 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201449.112275:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201449.112294:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201449.112476:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6a00/0x9ae3109cee9e143b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.112487:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6a00/0x9ae3109cee9e143b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.112497:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6a00/0x9ae3109cee9e143b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201449.112516:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe2680 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187373568:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201449.112771:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a639c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201449.115625:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe2680 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187373568:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201449.115719:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6a00/0x9ae3109cee9e143b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdf6a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.115756:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6a00/0x9ae3109cee9e143b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdf6a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.115768:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6a00/0x9ae3109cee9e143b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdf6a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.115778:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6a00/0x9ae3109cee9e143b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdf6a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.115790:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6a00/0x9ae3109cee9e143b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdf6a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.115798:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6a00/0x9ae3109cee9e143b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdf6a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.115805:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6a00/0x9ae3109cee9e143b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdf6a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201449.115817:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290fe2680 x1875062187373568/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201465 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201449.115831:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6a00/0x9ae3109cee9e143b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdf6a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.115844:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201449.115850:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201449.115877:0:41043:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201449.115890:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e143b 00000080:00010000:1.0:1788201449.115896:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201449.115898:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e143b from it ffffaa1800c97bf8 00010000:00010000:1.0:1788201449.115903:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6a00/0x9ae3109cee9e143b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdf6a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.115910:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6a00/0x9ae3109cee9e143b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdf6a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201449.115932:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201449.115955:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201449.115972:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201449.115980:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6a00/0x9ae3109cee9e143b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdf6a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201449.115989:0:41043:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6a00/0x9ae3109cee9e143b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdf6a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201449.116101:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201449.116108:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201449.116110:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e143b 00000080:00010000:1.0:1788201449.116115:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e143b from it ffffaa1800c97b38 00010000:00010000:1.0:1788201449.116118:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6a00/0x9ae3109cee9e143b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdf6a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201449.116125:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6a00/0x9ae3109cee9e143b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdf6a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201449.116147:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201449.116153:0:41043:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-2-84, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), file ffff8e82829b9980, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201449.116161:0:41043:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-2-84, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000002:00010000:1.0:1788201449.116184:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-2-84,[0x2000088d1:0x55:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201449.116196:0:41043:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6a00/0x9ae3109cee9e143b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbdf6a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201449.116251:0:41043:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6a00/0x9ae3109cee9e143b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbdf6a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201449.116258:0:41043:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201449.116261:0:41043:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6a00/0x9ae3109cee9e143b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbdf6a expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201449.116296:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7a00/0x9ae3109cee9e1442 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.116302:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7a00/0x9ae3109cee9e1442 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201449.116475:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201449.116483:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe2680 x1875062187373824/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201449.116492:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7a00/0x9ae3109cee9e1442 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201449.116514:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe2680 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187373824:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201449.116839:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a639c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201449.121122:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe2680 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187373824:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201449.121148:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7a00/0x9ae3109cee9e1442 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdf86 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.121177:0:41043:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7a00/0x9ae3109cee9e1442 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x55:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdf86 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.121186:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7a00/0x9ae3109cee9e1442 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdf86 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.121197:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7a00/0x9ae3109cee9e1442 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdf86 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.121208:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7a00/0x9ae3109cee9e1442 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdf86 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.121221:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7a00/0x9ae3109cee9e1442 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdf86 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.121230:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7a00/0x9ae3109cee9e1442 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdf86 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.121237:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7a00/0x9ae3109cee9e1442 lrc: 4/1,0 mode: PR/PR res: [0x2000088d1:0x55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbdf86 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201449.121249:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8290fe2680 x1875062187373824/t77309411597(77309411597) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201465 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201449.121277:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290fe2680 x1875062187373824/t77309411597(77309411597) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201465 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201449.121292:0:41043:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7a00/0x9ae3109cee9e1442 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdf86 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201449.121305:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7a00/0x9ae3109cee9e1442 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbdf86 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.121343:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589969617/85 (0 0) 00000080:00200000:1.0:1788201449.121349:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201449.121361:0:41043:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-2-84, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000080:00200000:1.0:1788201449.121395:0:41043:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000088d1:0x55:0x0](ffff8e82927f63c8) 00000080:00200000:1.0:1788201449.121468:0:41043:0:(file.c:6843:ll_layout_conf()) [0x2000088d1:0x55:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201449.121473:0:41043:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2000088d1:0x55:0x0](ffff8e82927f63c8) 00000080:00010000:1.0:1788201449.121477:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000088d1:0x55:0x0](ffff8e82927f63c8) for lock 0x9ae3109cee9e1442 00000080:00010000:1.0:1788201449.121486:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000088d1:0x55:0x0](ffff8e82927f63c8) 00000080:00200008:1.0:1788201449.121501:0:41043:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-2-84:[0x2000088d1:0x55:0x0](ffff8e82927f63c8), flags 100102 00000002:00100000:1.0:1788201449.121520:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290fe2680 x1875062187373824/t77309411597(77309411597) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201465 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201449.121548:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1442 from it ffff8e8287377240 00010000:00010000:1.0:1788201449.121553:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7a00/0x9ae3109cee9e1442 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbdf86 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.121563:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7a00/0x9ae3109cee9e1442 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x55:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbdf86 expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201449.121730:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000088d1:0x55:0x0](ffff8e82927f63c8), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201449.121769:0:41043:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-2-84:[0x2000088d1:0x55:0x0](ffff8e82927f63c8), flags 100002 00000002:00100000:1.0:1788201449.121814:0:41043:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8290fe2680 x1875062187373824/t77309411597(77309411597) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201465 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201449.121823:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201449.121828:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe0000 x1875062187374080/t0(0) o35->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201449.121848:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe0000 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187374080:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201449.121979:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201449.124739:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe0000 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187374080:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201449.124785:0:41043:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8290fe2680 x1875062187373824/t77309411597(77309411597) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201465 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201449.124859:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201449.124876:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201449.124930:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e1449 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.124940:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e1449 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.124949:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7e00/0x9ae3109cee9e1449 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201449.124968:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe2a00 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187374208:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201449.125237:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201449.253050:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894db480 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187377664:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201449.253078:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828b94aa00/0x9ae3109cee9e1496 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0ac expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.253107:0:41043:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828b94aa00/0x9ae3109cee9e1496 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x5b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0ac expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.253115:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828b94aa00/0x9ae3109cee9e1496 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x5b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0ac expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.253126:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828b94aa00/0x9ae3109cee9e1496 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x5b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe0ac expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.253136:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828b94aa00/0x9ae3109cee9e1496 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x5b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0ac expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.253148:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828b94aa00/0x9ae3109cee9e1496 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x5b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe0ac expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.253156:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828b94aa00/0x9ae3109cee9e1496 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x5b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe0ac expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.253162:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828b94aa00/0x9ae3109cee9e1496 lrc: 4/1,0 mode: PR/PR res: [0x2000088d1:0x5b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe0ac expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201449.253174:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e82894db480 x1875062187377664/t77309411609(77309411609) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201465 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201449.253202:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e82894db480 x1875062187377664/t77309411609(77309411609) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201465 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201449.253215:0:41043:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828b94aa00/0x9ae3109cee9e1496 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x5b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0ac expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201449.253228:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828b94aa00/0x9ae3109cee9e1496 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x5b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0ac expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.253244:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589969617/91 (0 0) 00000080:00200000:1.0:1788201449.253251:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201449.253263:0:41043:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-2-90, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000080:00200000:1.0:1788201449.253290:0:41043:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000088d1:0x5b:0x0](ffff8e82927f5b88) 00000080:00200000:1.0:1788201449.253377:0:41043:0:(file.c:6843:ll_layout_conf()) [0x2000088d1:0x5b:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201449.253383:0:41043:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2000088d1:0x5b:0x0](ffff8e82927f5b88) 00000080:00010000:1.0:1788201449.253386:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000088d1:0x5b:0x0](ffff8e82927f5b88) for lock 0x9ae3109cee9e1496 00000080:00010000:1.0:1788201449.253396:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000088d1:0x5b:0x0](ffff8e82927f5b88) 00000080:00200008:1.0:1788201449.253409:0:41043:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-2-90:[0x2000088d1:0x5b:0x0](ffff8e82927f5b88), flags 100102 00000002:00100000:1.0:1788201449.253429:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e82894db480 x1875062187377664/t77309411609(77309411609) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201465 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201449.253456:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1496 from it ffff8e8287ebb4e0 00010000:00010000:1.0:1788201449.253461:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828b94aa00/0x9ae3109cee9e1496 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x5b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbe0ac expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.253469:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828b94aa00/0x9ae3109cee9e1496 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x5b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbe0ac expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201449.253500:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000088d1:0x5b:0x0](ffff8e82927f5b88), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201449.253531:0:41043:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-2-90:[0x2000088d1:0x5b:0x0](ffff8e82927f5b88), flags 100002 00000002:00100000:1.0:1788201449.253590:0:41043:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e82894db480 x1875062187377664/t77309411609(77309411609) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201465 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201449.253601:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201449.253606:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe0380 x1875062187377920/t0(0) o35->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201449.253626:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe0380 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187377920:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201449.253827:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a636c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201449.262936:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe0380 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187377920:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201449.262974:0:41043:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e82894db480 x1875062187377664/t77309411609(77309411609) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201465 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201449.263056:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201449.263076:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201449.263133:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e149d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.263142:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e149d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.263153:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e149d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201449.263171:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe1180 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187378048:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201449.263273:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a636c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201449.266619:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe1180 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187378048:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201449.266647:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e149d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0c1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.266659:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e149d lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0c1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.266670:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e149d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe0c1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.266681:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e149d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0c1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.266694:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e149d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe0c1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.266702:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e149d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe0c1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.266709:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e149d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe0c1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201449.266721:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290fe1180 x1875062187378048/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201465 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201449.266735:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e149d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0c1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.266777:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201449.266784:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201449.266812:0:41043:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201449.266819:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e149d 00000080:00010000:1.0:1788201449.266825:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201449.266828:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e149d from it ffffaa1800c97bf8 00010000:00010000:1.0:1788201449.266832:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e149d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe0c1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.266841:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e149d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0c1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201449.266865:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201449.266890:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201449.266905:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201449.266915:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e149d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0c1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201449.266925:0:41043:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e149d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0c1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201449.266931:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201449.266935:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201449.266938:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e149d 00000080:00010000:1.0:1788201449.266942:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e149d from it ffffaa1800c97b38 00010000:00010000:1.0:1788201449.266945:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e149d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe0c1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201449.266960:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e149d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0c1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201449.266968:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201449.266974:0:41043:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-2-91, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), file ffff8e82829b9980, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201449.266983:0:41043:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-2-91, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000002:00010000:1.0:1788201449.267004:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-2-91,[0x2000088d1:0x5c:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201449.267016:0:41043:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e149d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbe0c1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201449.267073:0:41043:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e149d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe0c1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201449.267080:0:41043:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201449.267084:0:41043:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e149d lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe0c1 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201449.267116:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993800/0x9ae3109cee9e14a4 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.267123:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993800/0x9ae3109cee9e14a4 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201449.267132:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201449.267140:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe1180 x1875062187378304/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201449.267149:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993800/0x9ae3109cee9e14a4 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201449.267175:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe1180 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187378304:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201449.267356:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a636c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201449.272143:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe1180 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187378304:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201449.272172:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993800/0x9ae3109cee9e14a4 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0dd expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.272202:0:41043:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993800/0x9ae3109cee9e14a4 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x5c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0dd expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.272210:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993800/0x9ae3109cee9e14a4 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0dd expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.272220:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993800/0x9ae3109cee9e14a4 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe0dd expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.272229:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993800/0x9ae3109cee9e14a4 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0dd expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.272243:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993800/0x9ae3109cee9e14a4 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe0dd expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.272251:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993800/0x9ae3109cee9e14a4 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe0dd expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.272259:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993800/0x9ae3109cee9e14a4 lrc: 4/1,0 mode: PR/PR res: [0x2000088d1:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe0dd expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201449.272290:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8290fe1180 x1875062187378304/t77309411611(77309411611) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201465 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201449.272303:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290fe1180 x1875062187378304/t77309411611(77309411611) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201465 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201449.272360:0:41043:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993800/0x9ae3109cee9e14a4 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0dd expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201449.272373:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993800/0x9ae3109cee9e14a4 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0dd expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.272383:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589969617/92 (0 0) 00000080:00200000:1.0:1788201449.272389:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201449.272402:0:41043:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-2-91, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000080:00200000:1.0:1788201449.272430:0:41043:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000088d1:0x5c:0x0](ffff8e82927f3a88) 00000080:00200000:1.0:1788201449.272500:0:41043:0:(file.c:6843:ll_layout_conf()) [0x2000088d1:0x5c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201449.272505:0:41043:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2000088d1:0x5c:0x0](ffff8e82927f3a88) 00000080:00010000:1.0:1788201449.272508:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000088d1:0x5c:0x0](ffff8e82927f3a88) for lock 0x9ae3109cee9e14a4 00000080:00010000:1.0:1788201449.272517:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000088d1:0x5c:0x0](ffff8e82927f3a88) 00000080:00200008:1.0:1788201449.272530:0:41043:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-2-91:[0x2000088d1:0x5c:0x0](ffff8e82927f3a88), flags 100102 00000002:00100000:1.0:1788201449.272548:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290fe1180 x1875062187378304/t77309411611(77309411611) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201465 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201449.272589:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e14a4 from it ffff8e8287377000 00010000:00010000:1.0:1788201449.272594:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993800/0x9ae3109cee9e14a4 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbe0dd expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.272626:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993800/0x9ae3109cee9e14a4 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x5c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbe0dd expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201449.272656:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000088d1:0x5c:0x0](ffff8e82927f3a88), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201449.272687:0:41043:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-2-91:[0x2000088d1:0x5c:0x0](ffff8e82927f3a88), flags 100002 00000002:00100000:1.0:1788201449.272729:0:41043:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8290fe1180 x1875062187378304/t77309411611(77309411611) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201465 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201449.272738:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201449.272753:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe3800 x1875062187378560/t0(0) o35->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201449.272776:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe3800 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187378560:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201449.272849:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201449.275999:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe3800 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187378560:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201449.276035:0:41043:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8290fe1180 x1875062187378304/t77309411611(77309411611) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201465 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201449.276112:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201449.276129:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201449.276189:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14ab lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.276200:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14ab lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.276211:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14ab lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201449.276252:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe1880 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187378688:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201449.276370:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201449.283976:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe1880 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187378688:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201449.284002:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14ab lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0f2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.284014:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14ab lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0f2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.284025:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14ab lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe0f2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.284035:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14ab lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0f2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.284048:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14ab lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe0f2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.284056:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14ab lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe0f2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.284063:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14ab lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe0f2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201449.284075:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290fe1880 x1875062187378688/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201465 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201449.284109:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14ab lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0f2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.284121:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201449.284128:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201449.284154:0:41043:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201449.284160:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e14ab 00000080:00010000:1.0:1788201449.284165:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201449.284168:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e14ab from it ffffaa1800c97bf8 00010000:00010000:1.0:1788201449.284171:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14ab lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe0f2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.284181:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14ab lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0f2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201449.284203:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201449.284225:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201449.284240:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201449.284249:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14ab lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0f2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201449.284261:0:41043:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14ab lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0f2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201449.284268:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201449.284272:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201449.284275:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e14ab 00000080:00010000:1.0:1788201449.284285:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e14ab from it ffffaa1800c97b38 00010000:00010000:1.0:1788201449.284288:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14ab lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe0f2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201449.284295:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14ab lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe0f2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201449.284303:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201449.284353:0:41043:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-2-92, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), file ffff8e82829b8c00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201449.284382:0:41043:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-2-92, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000002:00010000:1.0:1788201449.284402:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-2-92,[0x2000088d1:0x5d:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201449.284415:0:41043:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14ab lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbe0f2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201449.284478:0:41043:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14ab lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe0f2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201449.284485:0:41043:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201449.284488:0:41043:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14ab lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe0f2 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201449.284519:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992c00/0x9ae3109cee9e14b2 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.284526:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992c00/0x9ae3109cee9e14b2 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201449.284536:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201449.284543:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe1880 x1875062187378944/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201449.284557:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992c00/0x9ae3109cee9e14b2 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201449.284589:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe1880 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187378944:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201449.284710:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201449.291379:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe1880 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187378944:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201449.291409:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992c00/0x9ae3109cee9e14b2 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe10e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.291451:0:41043:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992c00/0x9ae3109cee9e14b2 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x5d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe10e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.291459:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992c00/0x9ae3109cee9e14b2 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe10e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.291470:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992c00/0x9ae3109cee9e14b2 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe10e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.291479:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992c00/0x9ae3109cee9e14b2 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe10e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.291493:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992c00/0x9ae3109cee9e14b2 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe10e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.291502:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992c00/0x9ae3109cee9e14b2 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe10e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.291530:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992c00/0x9ae3109cee9e14b2 lrc: 4/1,0 mode: PR/PR res: [0x2000088d1:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe10e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201449.291542:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8290fe1880 x1875062187378944/t77309411613(77309411613) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201465 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201449.291554:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290fe1880 x1875062187378944/t77309411613(77309411613) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201465 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201449.291579:0:41043:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992c00/0x9ae3109cee9e14b2 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe10e expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201449.291593:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992c00/0x9ae3109cee9e14b2 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe10e expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.291603:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589969617/93 (0 0) 00000080:00200000:1.0:1788201449.291609:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201449.291623:0:41043:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-2-92, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000080:00200000:1.0:1788201449.291655:0:41043:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000088d1:0x5d:0x0](ffff8e82927f42c8) 00000080:00200000:1.0:1788201449.291729:0:41043:0:(file.c:6843:ll_layout_conf()) [0x2000088d1:0x5d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201449.291734:0:41043:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2000088d1:0x5d:0x0](ffff8e82927f42c8) 00000080:00010000:1.0:1788201449.291738:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000088d1:0x5d:0x0](ffff8e82927f42c8) for lock 0x9ae3109cee9e14b2 00000080:00010000:1.0:1788201449.291757:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000088d1:0x5d:0x0](ffff8e82927f42c8) 00000080:00200008:1.0:1788201449.291772:0:41043:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-2-92:[0x2000088d1:0x5d:0x0](ffff8e82927f42c8), flags 100102 00000002:00100000:1.0:1788201449.291792:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290fe1880 x1875062187378944/t77309411613(77309411613) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201465 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201449.291833:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e14b2 from it ffff8e82849b9480 00010000:00010000:1.0:1788201449.291837:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992c00/0x9ae3109cee9e14b2 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbe10e expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.291846:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992c00/0x9ae3109cee9e14b2 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x5d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbe10e expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201449.291866:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000088d1:0x5d:0x0](ffff8e82927f42c8), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201449.291897:0:41043:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-2-92:[0x2000088d1:0x5d:0x0](ffff8e82927f42c8), flags 100002 00000002:00100000:1.0:1788201449.291945:0:41043:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8290fe1880 x1875062187378944/t77309411613(77309411613) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201465 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201449.291955:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201449.291960:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe2d80 x1875062187379200/t0(0) o35->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201449.291978:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe2d80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187379200:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201449.292050:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201449.295161:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe2d80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187379200:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201449.295196:0:41043:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8290fe1880 x1875062187378944/t77309411613(77309411613) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201465 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201449.295279:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201449.295299:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201449.295374:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14b9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.295402:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14b9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.295413:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14b9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201449.295445:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe3480 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187379328:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201449.295523:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201449.299167:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe3480 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187379328:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201449.299194:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14b9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe123 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.299205:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14b9 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe123 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.299217:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14b9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe123 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.299228:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14b9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe123 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.299240:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14b9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe123 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.299247:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14b9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe123 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.299275:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14b9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe123 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201449.299288:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290fe3480 x1875062187379328/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201465 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201449.299302:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14b9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe123 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.299360:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201449.299367:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201449.299395:0:41043:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201449.299402:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e14b9 00000080:00010000:1.0:1788201449.299408:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201449.299411:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e14b9 from it ffffaa1800c97bf8 00010000:00010000:1.0:1788201449.299415:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14b9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe123 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.299424:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14b9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe123 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201449.299448:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201449.299471:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201449.299485:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201449.299494:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14b9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe123 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201449.299515:0:41043:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14b9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe123 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201449.299521:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201449.299526:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201449.299528:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e14b9 00000080:00010000:1.0:1788201449.299533:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e14b9 from it ffffaa1800c97b38 00010000:00010000:1.0:1788201449.299536:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14b9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe123 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201449.299543:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14b9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe123 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201449.299551:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201449.299556:0:41043:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-2-93, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), file ffff8e82829b9980, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201449.299564:0:41043:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-2-93, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000002:00010000:1.0:1788201449.299597:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-2-93,[0x2000088d1:0x5e:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201449.299609:0:41043:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14b9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbe123 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201449.299753:0:41043:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14b9 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe123 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201449.299761:0:41043:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201449.299765:0:41043:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14b9 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe123 expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201449.299810:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14c0 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.299817:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14c0 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201449.299827:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201449.299834:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe3480 x1875062187379584/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201449.299843:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14c0 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201449.299864:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe3480 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187379584:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201449.299985:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8288a63b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201449.309022:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe3480 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187379584:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201449.309050:0:41043:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14c0 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe13f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.309079:0:41043:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14c0 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x5e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe13f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.309088:0:41043:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14c0 lrc: 4/0,1 mode: --/PR res: [0x2000088d1:0x5e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe13f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.309098:0:41043:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14c0 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x5e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe13f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.309124:0:41043:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14c0 lrc: 4/0,1 mode: PR/PR res: [0x2000088d1:0x5e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe13f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.309138:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14c0 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x5e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe13f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.309146:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14c0 lrc: 5/1,1 mode: PR/PR res: [0x2000088d1:0x5e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe13f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.309153:0:41043:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14c0 lrc: 4/1,0 mode: PR/PR res: [0x2000088d1:0x5e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe13f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201449.309166:0:41043:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8290fe3480 x1875062187379584/t77309411615(77309411615) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201465 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201449.309178:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290fe3480 x1875062187379584/t77309411615(77309411615) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201465 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201449.309191:0:41043:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14c0 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x5e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe13f expref: -99 pid: 41043 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201449.309204:0:41043:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14c0 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x5e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe13f expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.309214:0:41043:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589969617/94 (0 0) 00000080:00200000:1.0:1788201449.309220:0:41043:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201449.309234:0:41043:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-2-93, dir=[0x200000007:0x1:0x0](ffff8e8289c29988), intent=open|creat 00000080:00200000:1.0:1788201449.309270:0:41043:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000088d1:0x5e:0x0](ffff8e8289c15348) 00000080:00200000:1.0:1788201449.309374:0:41043:0:(file.c:6843:ll_layout_conf()) [0x2000088d1:0x5e:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201449.309379:0:41043:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x2000088d1:0x5e:0x0](ffff8e8289c15348) 00000080:00010000:1.0:1788201449.309382:0:41043:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000088d1:0x5e:0x0](ffff8e8289c15348) for lock 0x9ae3109cee9e14c0 00000080:00010000:1.0:1788201449.309392:0:41043:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000088d1:0x5e:0x0](ffff8e8289c15348) 00000080:00200008:1.0:1788201449.309406:0:41043:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-2-93:[0x2000088d1:0x5e:0x0](ffff8e8289c15348), flags 100102 00000002:00100000:1.0:1788201449.309426:0:41043:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290fe3480 x1875062187379584/t77309411615(77309411615) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201465 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201449.309453:0:41043:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e14c0 from it ffff8e82849b9480 00010000:00010000:1.0:1788201449.309457:0:41043:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14c0 lrc: 3/1,0 mode: PR/PR res: [0x2000088d1:0x5e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbe13f expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.309466:0:41043:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993a00/0x9ae3109cee9e14c0 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x5e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbe13f expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201449.309498:0:41043:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000088d1:0x5e:0x0](ffff8e8289c15348), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201449.309526:0:41043:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-2-93:[0x2000088d1:0x5e:0x0](ffff8e8289c15348), flags 100002 00000002:00100000:1.0:1788201449.309566:0:41043:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8290fe3480 x1875062187379584/t77309411615(77309411615) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201465 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201449.309585:0:41043:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8284ef2800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201449.309590:0:41043:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe3100 x1875062187379840/t0(0) o35->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201449.309611:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe3100 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187379840:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201449.309689:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8299c0dd80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201449.312374:0:41043:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe3100 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187379840:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201449.312444:0:41043:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8290fe3480 x1875062187379584/t77309411615(77309411615) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201465 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201449.312523:0:41043:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201449.312541:0:41043:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201449.312598:0:41043:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14c7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.312607:0:41043:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14c7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201449.312617:0:41043:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287993200/0x9ae3109cee9e14c7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41043 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201449.312637:0:41043:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe1f80 pname:cluuid:pid:xid:nid:opc:job createmany:6fe158f2-55b2-4a06-bd78-c92321304d95:41043:1875062187379968:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201449.312714:0:41043:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8299c0dd80 going to sleep for 11 seconds 00000080:00200008:1.0:1788201450.223332:0:41052:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8e8292531988), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201450.223348:0:41052:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201450.223373:0:41052:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201450.223434:0:41052:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e151b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41052 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.223445:0:41052:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e151b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41052 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.223455:0:41052:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e151b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41052 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201450.223479:0:41052:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe0e00 pname:cluuid:pid:xid:nid:opc:job lfs:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41052:1875062187383808:192.168.201.121@tcp:101:lfs.0 00000100:00100000:1.0:1788201450.223575:0:41052:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8299c0dd80 going to sleep for 11 seconds 00000080:00200080:0.0:1788201450.275191:0:41052:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe03412390 00020000:00000080:0.0:1788201450.275201:0:41052:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290989800 uarg=0000000000000000 00000080:00200008:0.0:1788201450.275216:0:41052:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8e8292531988), flags 100000 00000080:00200000:1.0:1788201450.371293:0:41062:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201450.371342:0:41062:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201450.371402:0:41062:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1522 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41062 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.371413:0:41062:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1522 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41062 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.371425:0:41062:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1522 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41062 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201450.371450:0:41062:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe0e00 pname:cluuid:pid:xid:nid:opc:job rm:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41062:1875062187384704:192.168.201.121@tcp:101:rm.0 00000100:00100000:1.0:1788201450.371547:0:41062:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8299c0dd80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201450.377820:0:41062:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe0e00 pname:cluuid:pid:xid:nid:opc:job rm:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41062:1875062187384704:192.168.201.121@tcp:101:rm.0 00010000:00010000:1.0:1788201450.377848:0:41062:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1522 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe296 expref: -99 pid: 41062 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.377928:0:41062:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1522 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe296 expref: -99 pid: 41062 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.377942:0:41062:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1522 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe296 expref: -99 pid: 41062 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.377953:0:41062:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1522 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe296 expref: -99 pid: 41062 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.377966:0:41062:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1522 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe296 expref: -99 pid: 41062 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.378177:0:41062:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1522 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe296 expref: -99 pid: 41062 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.378185:0:41062:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1522 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe296 expref: -99 pid: 41062 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201450.378197:0:41062:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290fe0e00 x1875062187384704/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201466 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201450.378210:0:41062:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1522 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe296 expref: -99 pid: 41062 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.378222:0:41062:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201450.378229:0:41062:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201450.379128:0:41062:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:0.0:1788201450.379140:0:41062:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1522 00000080:00010000:0.0:1788201450.379147:0:41062:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201450.379150:0:41062:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1522 from it ffffaa1801fa7b58 00010000:00010000:0.0:1788201450.379154:0:41062:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1522 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe296 expref: -99 pid: 41062 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201450.379165:0:41062:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1522 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe296 expref: -99 pid: 41062 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201450.379189:0:41062:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201450.379207:0:41062:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg121-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=8 00000080:00200000:0.0:1788201450.379211:0:41062:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg121-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201450.379231:0:41062:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: fsa-oleg121-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201450.379274:0:41062:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1529 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41062 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201450.379282:0:41062:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1529 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41062 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201450.379291:0:41062:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1529 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41062 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201450.379416:0:41062:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b90380 pname:cluuid:pid:xid:nid:opc:job rm:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41062:1875062187384832:192.168.201.121@tcp:101:rm.0 00000100:00100000:0.0:1788201450.379534:0:41062:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287be0c00 going to sleep for 11 seconds 00000080:00200008:0.0:1788201450.451095:0:41066:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8e8289c29988), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201450.451116:0:41066:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:0.0:1788201450.451139:0:41066:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201450.451202:0:41066:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1200/0x9ae3109cee9e1530 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41066 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201450.451212:0:41066:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1200/0x9ae3109cee9e1530 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41066 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201450.451222:0:41066:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1200/0x9ae3109cee9e1530 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41066 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201450.451246:0:41066:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b91c00 pname:cluuid:pid:xid:nid:opc:job bash:6fe158f2-55b2-4a06-bd78-c92321304d95:41066:1875062187385216:192.168.201.121@tcp:101:bash.0 00000100:00100000:0.0:1788201450.451360:0:41066:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287be0900 going to sleep for 11 seconds 00000100:00100000:1.0:1788201450.461399:0:41066:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8284ef2800: committing for last_committed 77309411628 gen 1 00000100:00100000:1.0:1788201450.463536:0:41066:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b91c00 pname:cluuid:pid:xid:nid:opc:job bash:6fe158f2-55b2-4a06-bd78-c92321304d95:41066:1875062187385216:192.168.201.121@tcp:101:bash.0 00010000:00010000:1.0:1788201450.463563:0:41066:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1200/0x9ae3109cee9e1530 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe2dc expref: -99 pid: 41066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.463575:0:41066:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1200/0x9ae3109cee9e1530 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe2dc expref: -99 pid: 41066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.463587:0:41066:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1200/0x9ae3109cee9e1530 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe2dc expref: -99 pid: 41066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.463598:0:41066:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1200/0x9ae3109cee9e1530 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe2dc expref: -99 pid: 41066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.463626:0:41066:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1200/0x9ae3109cee9e1530 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe2dc expref: -99 pid: 41066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.463635:0:41066:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1200/0x9ae3109cee9e1530 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe2dc expref: -99 pid: 41066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.463643:0:41066:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1200/0x9ae3109cee9e1530 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe2dc expref: -99 pid: 41066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201450.463654:0:41066:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8288b91c00 x1875062187385216/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201466 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201450.463668:0:41066:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1200/0x9ae3109cee9e1530 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe2dc expref: -99 pid: 41066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.463680:0:41066:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201450.463686:0:41066:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201450.463715:0:41066:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201450.463734:0:41066:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) for lock 0x9ae3109cee9e1530 00000080:00010000:1.0:1788201450.463739:0:41066:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c29988) 00000080:00010000:1.0:1788201450.463742:0:41066:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1530 from it ffffaa1801fb7cb8 00010000:00010000:1.0:1788201450.463746:0:41066:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1200/0x9ae3109cee9e1530 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe2dc expref: -99 pid: 41066 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.463756:0:41066:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1200/0x9ae3109cee9e1530 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe2dc expref: -99 pid: 41066 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201450.516468:0:41072:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201450.516499:0:41072:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201450.516562:0:41072:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992600/0x9ae3109cee9e1537 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41072 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.516572:0:41072:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992600/0x9ae3109cee9e1537 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41072 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.516582:0:41072:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992600/0x9ae3109cee9e1537 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41072 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201450.516772:0:41072:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe3100 pname:cluuid:pid:xid:nid:opc:job rm:6fe158f2-55b2-4a06-bd78-c92321304d95:41072:1875062187385600:192.168.201.121@tcp:101:rm.0 00000100:00100000:1.0:1788201450.516863:0:41072:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8299c0dd80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201450.533213:0:41072:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe3100 pname:cluuid:pid:xid:nid:opc:job rm:6fe158f2-55b2-4a06-bd78-c92321304d95:41072:1875062187385600:192.168.201.121@tcp:101:rm.0 00010000:00010000:1.0:1788201450.533239:0:41072:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992600/0x9ae3109cee9e1537 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe2f8 expref: -99 pid: 41072 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.533251:0:41072:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992600/0x9ae3109cee9e1537 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe2f8 expref: -99 pid: 41072 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.533262:0:41072:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992600/0x9ae3109cee9e1537 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe2f8 expref: -99 pid: 41072 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.533273:0:41072:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992600/0x9ae3109cee9e1537 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe2f8 expref: -99 pid: 41072 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.533286:0:41072:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992600/0x9ae3109cee9e1537 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe2f8 expref: -99 pid: 41072 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.533399:0:41072:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992600/0x9ae3109cee9e1537 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe2f8 expref: -99 pid: 41072 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.533408:0:41072:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992600/0x9ae3109cee9e1537 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe2f8 expref: -99 pid: 41072 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201450.533420:0:41072:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290fe3100 x1875062187385600/t0(0) o101->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201466 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201450.533433:0:41072:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992600/0x9ae3109cee9e1537 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe2f8 expref: -99 pid: 41072 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201450.533445:0:41072:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8290f20c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201450.533452:0:41072:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000100:00100000:0.0:1788201452.038935:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187386112:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201452.039034:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187386368:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201452.039060:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187386624:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201452.039083:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6fe158f2-55b2-4a06-bd78-c92321304d95:2401:1875062187386880:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201452.039109:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6fe158f2-55b2-4a06-bd78-c92321304d95:2401:1875062187387136:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201452.039140:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [1->0] 00000100:00100000:0.0:1788201452.039146:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187386240:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201452.039171:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187386496:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201452.039205:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6fe158f2-55b2-4a06-bd78-c92321304d95:2401:1875062187386752:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201452.039225:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6fe158f2-55b2-4a06-bd78-c92321304d95:2401:1875062187387008:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201452.045547:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187386112:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201452.045568:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187386368:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201452.045579:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6fe158f2-55b2-4a06-bd78-c92321304d95:2401:1875062187386880:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201452.067614:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187386240:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201452.067626:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187386496:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201452.067634:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6fe158f2-55b2-4a06-bd78-c92321304d95:2401:1875062187386752:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201452.067648:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6fe158f2-55b2-4a06-bd78-c92321304d95:2401:1875062187387008:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201452.067656:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187386624:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201452.067666:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:6fe158f2-55b2-4a06-bd78-c92321304d95:2401:1875062187387136:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201457.156816:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187387264:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201457.156939:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187387520:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201457.156982:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187387776:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201457.157003:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6fe158f2-55b2-4a06-bd78-c92321304d95:2402:1875062187388032:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201457.157021:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6fe158f2-55b2-4a06-bd78-c92321304d95:2402:1875062187388288:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201457.157047:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [0->1] 00000100:00100000:1.0:1788201457.157053:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b92a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187387392:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201457.157075:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187387648:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201457.157094:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6fe158f2-55b2-4a06-bd78-c92321304d95:2402:1875062187387904:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201457.157112:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6fe158f2-55b2-4a06-bd78-c92321304d95:2402:1875062187388160:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201457.164558:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b92a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187387392:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201457.164595:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b91c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187387648:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201457.164619:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b91180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6fe158f2-55b2-4a06-bd78-c92321304d95:2402:1875062187387904:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201457.164636:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b93800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6fe158f2-55b2-4a06-bd78-c92321304d95:2402:1875062187388160:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201457.164647:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b91880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187387264:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201457.164661:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b92680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187387520:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201457.164674:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b91500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187387776:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201457.164687:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b90e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6fe158f2-55b2-4a06-bd78-c92321304d95:2402:1875062187388032:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201457.164715:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b92300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:6fe158f2-55b2-4a06-bd78-c92321304d95:2402:1875062187388288:192.168.201.121@tcp:400:kworker.0 00000080:00200000:0.0:1788201457.555741:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201457.555769:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201457.555836:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1545 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.555847:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1545 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.555857:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1545 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201457.555879:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b93480 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187388416:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201457.555958:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287be0c00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201457.560950:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b93480 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187388416:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201457.560974:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1545 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe33e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.560986:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1545 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe33e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.561009:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1545 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe33e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.561020:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1545 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe33e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.561032:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1545 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe33e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.561040:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1545 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe33e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.561047:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1545 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe33e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201457.561060:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8288b93480 x1875062187388416/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201473 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201457.561073:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1545 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe33e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.561101:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201457.561108:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201457.561134:0:41142:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201457.561141:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1545 00000080:00010000:1.0:1788201457.561147:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201457.561151:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1545 from it ffffaa1801fcfbf8 00010000:00010000:1.0:1788201457.561154:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1545 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe33e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.561163:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1545 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe33e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201457.561188:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201457.561207:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201457.561222:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201457.561231:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1545 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe33e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201457.561241:0:41142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1545 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe33e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201457.561247:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201457.561252:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201457.561255:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1545 00000080:00010000:1.0:1788201457.561258:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1545 from it ffffaa1801fcfb38 00010000:00010000:1.0:1788201457.561261:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1545 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe33e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201457.561277:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1545 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe33e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201457.561286:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201457.561292:0:41142:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-1-0, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8282991380, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201457.561300:0:41142:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-0, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:1.0:1788201457.561335:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-0,[0x200006991:0x73:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201457.561348:0:41142:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1545 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbe33e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201457.561389:0:41142:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1545 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe33e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201457.561396:0:41142:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201457.561400:0:41142:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1545 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe33e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201457.561434:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e154c lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.561441:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e154c lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201457.561450:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201457.561456:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe1500 x1875062187388672/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201457.561464:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e154c lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201457.561489:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe1500 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187388672:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201457.561619:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8299c0dd80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201457.655857:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d9180 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187389312:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201457.655887:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e155a lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe38b expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.655919:0:41142:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e155a lrc: 4/0,1 mode: --/PR res: [0x200006991:0x74:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe38b expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.655929:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e155a lrc: 4/0,1 mode: --/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe38b expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.655942:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e155a lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe38b expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.655954:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e155a lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe38b expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.655967:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e155a lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe38b expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.655976:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e155a lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe38b expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.655985:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e155a lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe38b expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201457.656012:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e82894d9180 x1875062187389312/t77309411635(77309411635) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201473 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201457.656026:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e82894d9180 x1875062187389312/t77309411635(77309411635) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201473 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201457.656042:0:41142:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e155a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe38b expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201457.656055:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e155a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe38b expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.656066:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/116 (0 0) 00000080:00200000:1.0:1788201457.656073:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201457.656086:0:41142:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-1-1, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:1.0:1788201457.656117:0:41142:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0x74:0x0](ffff8e8289c142c8) 00000080:00200000:1.0:1788201457.656210:0:41142:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x74:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201457.656215:0:41142:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0x74:0x0](ffff8e8289c142c8) 00000080:00010000:1.0:1788201457.656219:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x74:0x0](ffff8e8289c142c8) for lock 0x9ae3109cee9e155a 00000080:00010000:1.0:1788201457.656228:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x74:0x0](ffff8e8289c142c8) 00000080:00200008:1.0:1788201457.656242:0:41142:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-1-1:[0x200006991:0x74:0x0](ffff8e8289c142c8), flags 100102 00000002:00100000:1.0:1788201457.656263:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e82894d9180 x1875062187389312/t77309411635(77309411635) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201473 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201457.656329:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e155a from it ffff8e8287ebb8a0 00010000:00010000:1.0:1788201457.656335:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e155a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbe38b expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.656355:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e155a lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbe38b expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201457.656375:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0x74:0x0](ffff8e8289c142c8), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201457.656403:0:41142:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-1-1:[0x200006991:0x74:0x0](ffff8e8289c142c8), flags 100002 00000002:00100000:1.0:1788201457.656455:0:41142:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e82894d9180 x1875062187389312/t77309411635(77309411635) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201473 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201457.656465:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201457.656471:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe1180 x1875062187389568/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201457.656491:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe1180 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187389568:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201457.656558:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1000 going to sleep for 11 seconds 00000100:00100000:1.0:1788201457.685965:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894da300 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187389696:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201457.685997:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1561 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe3a0 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.686010:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1561 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe3a0 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.686024:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1561 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe3a0 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.686036:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1561 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe3a0 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.686063:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1561 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe3a0 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.686071:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1561 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe3a0 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.686080:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1561 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe3a0 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201457.686092:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894da300 x1875062187389696/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201473 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201457.686105:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1561 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe3a0 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.686118:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201457.686125:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201457.686154:0:41142:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201457.686236:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1561 00000080:00010000:1.0:1788201457.686244:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201457.686248:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1561 from it ffffaa1801fcfbf8 00010000:00010000:1.0:1788201457.686251:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1561 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe3a0 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.686260:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1561 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe3a0 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201457.686381:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201457.686403:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201457.686420:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201457.686429:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1561 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe3a0 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201457.686440:0:41142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1561 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe3a0 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201457.686447:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201457.686452:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201457.686455:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1561 00000080:00010000:1.0:1788201457.686459:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1561 from it ffffaa1801fcfb38 00010000:00010000:1.0:1788201457.686462:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1561 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe3a0 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201457.686469:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1561 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe3a0 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201457.686477:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201457.686483:0:41142:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-1-2, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8287b36000, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201457.686492:0:41142:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-2, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:1.0:1788201457.686511:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-2,[0x200006991:0x75:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201457.686522:0:41142:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1561 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbe3a0 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201457.686581:0:41142:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1561 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe3a0 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201457.686587:0:41142:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201457.686590:0:41142:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1561 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe3a0 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201457.686639:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e1568 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.686646:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e1568 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201457.686657:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201457.686664:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe2680 x1875062187389952/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201457.686673:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e1568 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201457.688805:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe2680 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187389952:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201457.701658:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1300 going to sleep for 11 seconds 00000100:00100000:0.0:1788201457.716624:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894db480 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187390208:192.168.201.121@tcp:35:createmany.0 00000002:00100000:0.0:1788201457.716662:0:41142:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8290fe2680 x1875062187389952/t77309411637(77309411637) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201473 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201457.717186:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201457.717205:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201457.717264:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e156f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.717274:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e156f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.717291:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e156f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201457.717329:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b93100 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187390336:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201457.717392:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287be0c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201457.730921:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b93100 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187390336:192.168.201.121@tcp:101:createmany.0 00010000:00010000:0.0:1788201457.730949:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e156f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe3d1 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.730962:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e156f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe3d1 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.730973:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e156f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe3d1 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.730983:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e156f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe3d1 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.730997:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e156f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe3d1 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.731004:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e156f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe3d1 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.731011:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e156f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe3d1 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201457.731043:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8288b93100 x1875062187390336/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201473 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201457.731056:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e156f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe3d1 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.731079:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201457.731086:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201457.731211:0:41142:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201457.731220:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e156f 00000080:00010000:1.0:1788201457.731225:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201457.731228:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e156f from it ffffaa1801fcfbf8 00010000:00010000:1.0:1788201457.731231:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e156f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe3d1 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.731251:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e156f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe3d1 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201457.731285:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201457.731326:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201457.731340:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201457.731349:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e156f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe3d1 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201457.731361:0:41142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e156f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe3d1 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201457.731368:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201457.731372:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201457.731375:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e156f 00000080:00010000:1.0:1788201457.731379:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e156f from it ffffaa1801fcfb38 00010000:00010000:1.0:1788201457.731382:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e156f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe3d1 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201457.731389:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e156f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe3d1 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201457.731397:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201457.731403:0:41142:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-1-3, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8282990f00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201457.731410:0:41142:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-3, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:1.0:1788201457.731493:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-3,[0x200006991:0x76:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201457.731571:0:41142:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e156f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbe3d1 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201457.731629:0:41142:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e156f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe3d1 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201457.731636:0:41142:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201457.731639:0:41142:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e156f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe3d1 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201457.731675:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1576 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.731683:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1576 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201457.731694:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201457.731702:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe0380 x1875062187390592/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201457.731710:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1576 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201457.731732:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe0380 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187390592:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201457.731996:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201457.812518:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894db100 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187391872:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201457.812549:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e1592 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe44f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.812579:0:41142:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e1592 lrc: 4/0,1 mode: --/PR res: [0x200006991:0x78:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe44f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.812609:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e1592 lrc: 4/0,1 mode: --/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe44f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.812622:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e1592 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe44f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.812634:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e1592 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe44f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.812648:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e1592 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe44f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.812658:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e1592 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe44f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.812667:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e1592 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe44f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201457.812679:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e82894db100 x1875062187391872/t77309411643(77309411643) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201473 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201457.812693:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e82894db100 x1875062187391872/t77309411643(77309411643) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201473 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201457.812706:0:41142:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e1592 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe44f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201457.812720:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e1592 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe44f expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.812736:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/120 (0 0) 00000080:00200000:1.0:1788201457.812743:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201457.812757:0:41142:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-1-5, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:1.0:1788201457.812785:0:41142:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0x78:0x0](ffff8e8289c13a88) 00000080:00200000:1.0:1788201457.812864:0:41142:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x78:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201457.812869:0:41142:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0x78:0x0](ffff8e8289c13a88) 00000080:00010000:1.0:1788201457.812872:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x78:0x0](ffff8e8289c13a88) for lock 0x9ae3109cee9e1592 00000080:00010000:1.0:1788201457.812881:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x78:0x0](ffff8e8289c13a88) 00000080:00200008:1.0:1788201457.812896:0:41142:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-1-5:[0x200006991:0x78:0x0](ffff8e8289c13a88), flags 100102 00000002:00100000:1.0:1788201457.812916:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e82894db100 x1875062187391872/t77309411643(77309411643) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201473 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201457.812948:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1592 from it ffff8e8287ebb900 00010000:00010000:1.0:1788201457.812954:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e1592 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbe44f expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.812964:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e1592 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbe44f expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201457.812982:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0x78:0x0](ffff8e8289c13a88), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201457.813008:0:41142:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-1-5:[0x200006991:0x78:0x0](ffff8e8289c13a88), flags 100002 00000002:00100000:1.0:1788201457.813050:0:41142:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e82894db100 x1875062187391872/t77309411643(77309411643) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201473 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201457.813058:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201457.813062:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe3800 x1875062187392128/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201457.813409:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe3800 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187392128:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201457.813485:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1600 going to sleep for 11 seconds 00000100:00100000:1.0:1788201457.926252:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.121@tcp (source: 12345-192.168.201.121@tcp) 00000100:00100000:0.0:1788201457.926360:0:6162:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875062211973504 00000100:00100000:0.0:1788201457.926369:0:6162:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875062211973504 00000100:00100000:0.0:1788201457.926417:0:6162:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.121@tcp, seq: 92 00000100:00100000:0.0:1788201457.926426:0:6162:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8e8290fe2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:6fe158f2-55b2-4a06-bd78-c92321304d95+5:6520:x1875062211973504:12345-192.168.201.121@tcp:104: 00010000:00010000:0.0:1788201457.926444:0:6162:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828b94b800/0x9ae3109cee9e15d8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x7cf2114380cbe536 expref: -99 pid: 41153 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201457.926518:0:6162:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788201457.926526:0:6162:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8e8290fe2d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:6fe158f2-55b2-4a06-bd78-c92321304d95+5:6520:x1875062211973504:12345-192.168.201.121@tcp:104: Request processed in 101us (284us total) trans 0 rc 0/0 00000100:00100000:0.0:1788201457.926533:0:6162:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.121@tcp, seq: 92 00000100:00100000:1.0:1788201457.942925:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d8380 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187395712:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201457.942954:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94ac00/0x9ae3109cee9e15e6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe567 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.942966:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94ac00/0x9ae3109cee9e15e6 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe567 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.942980:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94ac00/0x9ae3109cee9e15e6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe567 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.942992:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94ac00/0x9ae3109cee9e15e6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe567 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.943016:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94ac00/0x9ae3109cee9e15e6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe567 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.943025:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94ac00/0x9ae3109cee9e15e6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe567 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.943033:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94ac00/0x9ae3109cee9e15e6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe567 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201457.943045:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d8380 x1875062187395712/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201473 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201457.943077:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94ac00/0x9ae3109cee9e15e6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe567 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.943091:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201457.943097:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201457.943123:0:41142:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201457.943130:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e15e6 00000080:00010000:1.0:1788201457.943134:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201457.943138:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e15e6 from it ffffaa1801fcfbf8 00010000:00010000:1.0:1788201457.943142:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94ac00/0x9ae3109cee9e15e6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe567 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.943152:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94ac00/0x9ae3109cee9e15e6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe567 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201457.943177:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201457.943197:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201457.943211:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201457.943219:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94ac00/0x9ae3109cee9e15e6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe567 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201457.943230:0:41142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94ac00/0x9ae3109cee9e15e6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe567 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201457.943237:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201457.943241:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201457.943244:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e15e6 00000080:00010000:1.0:1788201457.943254:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e15e6 from it ffffaa1801fcfb38 00010000:00010000:1.0:1788201457.943257:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94ac00/0x9ae3109cee9e15e6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe567 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201457.943266:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94ac00/0x9ae3109cee9e15e6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe567 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201457.943275:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201457.943280:0:41142:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-1-11, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8287b36780, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201457.943288:0:41142:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-11, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:1.0:1788201457.943348:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-11,[0x200006991:0x7e:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201457.943360:0:41142:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94ac00/0x9ae3109cee9e15e6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbe567 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201457.943407:0:41142:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94ac00/0x9ae3109cee9e15e6 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe567 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201457.943414:0:41142:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201457.944744:0:41142:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94ac00/0x9ae3109cee9e15e6 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe567 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201457.944800:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e15ed lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201457.944807:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e15ed lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201457.944830:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201457.944841:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe3b80 x1875062187395968/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201457.944860:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e15ed lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201457.944882:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe3b80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187395968:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201457.945124:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c16c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201457.971444:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894da680 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187396864:192.168.201.121@tcp:35:createmany.0 00000002:00100000:0.0:1788201457.971479:0:41142:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e82894d9880 x1875062187396608/t77309411657(77309411657) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201473 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201457.971580:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201457.971609:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201457.971668:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1602 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.971697:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1602 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.971710:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1602 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201457.971730:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b90a80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187396992:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201457.971901:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287be0900 going to sleep for 11 seconds 00010000:00010000:0.0:1788201457.988505:0:41142:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1609 lrc: 4/0,1 mode: --/PR res: [0x200006991:0x80:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe5e5 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.988514:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1609 lrc: 4/0,1 mode: --/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe5e5 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.988525:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1609 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe5e5 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.988536:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1609 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe5e5 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.988549:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1609 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe5e5 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.988557:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1609 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe5e5 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.988565:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1609 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe5e5 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201457.988586:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e82894d8380 x1875062187397248/t77309411659(77309411659) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201473 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201457.988610:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e82894d8380 x1875062187397248/t77309411659(77309411659) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201473 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201457.988625:0:41142:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1609 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe5e5 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201457.988638:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1609 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe5e5 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.988649:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/128 (0 0) 00000080:00200000:0.0:1788201457.988656:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201457.988669:0:41142:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-1-13, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:0.0:1788201457.988692:0:41142:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0x80:0x0](ffff8e8289c0c2c8) 00000080:00200000:0.0:1788201457.988760:0:41142:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x80:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201457.988764:0:41142:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0x80:0x0](ffff8e8289c0c2c8) 00000080:00010000:0.0:1788201457.988768:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x80:0x0](ffff8e8289c0c2c8) for lock 0x9ae3109cee9e1609 00000080:00010000:0.0:1788201457.988777:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x80:0x0](ffff8e8289c0c2c8) 00000080:00200008:0.0:1788201457.988789:0:41142:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-1-13:[0x200006991:0x80:0x0](ffff8e8289c0c2c8), flags 100102 00000002:00100000:0.0:1788201457.988886:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e82894d8380 x1875062187397248/t77309411659(77309411659) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201473 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201457.988918:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1609 from it ffff8e8287ebb5a0 00010000:00010000:0.0:1788201457.988923:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1609 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbe5e5 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201457.988942:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1609 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbe5e5 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201457.989180:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0x80:0x0](ffff8e8289c0c2c8), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201457.989203:0:41142:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-1-13:[0x200006991:0x80:0x0](ffff8e8289c0c2c8), flags 100002 00000002:00100000:0.0:1788201457.989243:0:41142:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e82894d8380 x1875062187397248/t77309411659(77309411659) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201473 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201457.989253:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201457.989259:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8288b93b80 x1875062187397504/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201457.989279:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b93b80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187397504:192.168.201.121@tcp:35:createmany.0 00000100:00100000:0.0:1788201457.989454:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287be0900 going to sleep for 11 seconds 00000100:00100000:1.0:1788201458.051322:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288bafb80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187399552:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201458.051348:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204a200/0x9ae3109cee9e163a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe68d expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.051361:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204a200/0x9ae3109cee9e163a lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe68d expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.051372:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204a200/0x9ae3109cee9e163a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe68d expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.051383:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204a200/0x9ae3109cee9e163a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe68d expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.051396:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204a200/0x9ae3109cee9e163a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe68d expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.051405:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204a200/0x9ae3109cee9e163a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe68d expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.051412:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204a200/0x9ae3109cee9e163a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe68d expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201458.051448:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8288bafb80 x1875062187399552/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201458.051463:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204a200/0x9ae3109cee9e163a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe68d expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.051474:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201458.051481:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201458.051512:0:41142:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 12288 00000080:00010000:1.0:1788201458.051518:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e163a 00000080:00010000:1.0:1788201458.051523:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201458.051526:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e163a from it ffffaa1801fcfbf8 00010000:00010000:1.0:1788201458.051530:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204a200/0x9ae3109cee9e163a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe68d expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.051540:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204a200/0x9ae3109cee9e163a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe68d expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201458.051565:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201458.051580:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201458.051603:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201458.051613:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204a200/0x9ae3109cee9e163a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe68d expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201458.051624:0:41142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204a200/0x9ae3109cee9e163a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe68d expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201458.051641:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201458.051645:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201458.051649:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e163a 00000080:00010000:1.0:1788201458.051653:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e163a from it ffffaa1801fcfb38 00010000:00010000:1.0:1788201458.051656:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204a200/0x9ae3109cee9e163a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe68d expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201458.051664:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204a200/0x9ae3109cee9e163a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe68d expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201458.051673:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201458.051678:0:41142:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-1-17, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8287b36600, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201458.051685:0:41142:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-17, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:1.0:1788201458.051702:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-17,[0x200006991:0x84:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201458.051713:0:41142:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204a200/0x9ae3109cee9e163a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbe68d expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201458.051754:0:41142:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204a200/0x9ae3109cee9e163a lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe68d expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201458.051761:0:41142:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201458.051764:0:41142:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204a200/0x9ae3109cee9e163a lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe68d expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201458.051794:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1641 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.051810:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1641 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201458.051832:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201458.051839:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe0a80 x1875062187399808/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201458.051849:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1641 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201458.051869:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe0a80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187399808:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201458.051991:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1c00 going to sleep for 11 seconds 00000100:00100000:1.0:1788201458.064370:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe0a80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187399808:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201458.064399:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1641 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe6a9 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.064435:0:41142:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1641 lrc: 4/0,1 mode: --/PR res: [0x200006991:0x84:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe6a9 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.064444:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1641 lrc: 4/0,1 mode: --/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe6a9 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.064454:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1641 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe6a9 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.064463:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1641 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe6a9 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.064708:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1641 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe6a9 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.064720:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1641 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe6a9 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.064727:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1641 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe6a9 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201458.064740:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8290fe0a80 x1875062187399808/t77309411667(77309411667) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201458.064752:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290fe0a80 x1875062187399808/t77309411667(77309411667) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201458.064766:0:41142:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1641 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe6a9 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201458.064779:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1641 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe6a9 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.064790:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/132 (0 0) 00000080:00200000:1.0:1788201458.064796:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201458.064810:0:41142:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-1-17, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:1.0:1788201458.064847:0:41142:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0x84:0x0](ffff8e8289c163c8) 00000080:00200000:1.0:1788201458.064936:0:41142:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x84:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201458.064941:0:41142:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0x84:0x0](ffff8e8289c163c8) 00000080:00010000:1.0:1788201458.064948:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x84:0x0](ffff8e8289c163c8) for lock 0x9ae3109cee9e1641 00000080:00010000:1.0:1788201458.065174:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x84:0x0](ffff8e8289c163c8) 00000080:00200008:1.0:1788201458.065189:0:41142:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-1-17:[0x200006991:0x84:0x0](ffff8e8289c163c8), flags 100102 00000002:00100000:1.0:1788201458.065212:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290fe0a80 x1875062187399808/t77309411667(77309411667) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201458.065244:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1641 from it ffff8e82849b94e0 00010000:00010000:1.0:1788201458.065249:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1641 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbe6a9 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.065259:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1641 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbe6a9 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201458.065280:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0x84:0x0](ffff8e8289c163c8), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201458.065305:0:41142:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-1-17:[0x200006991:0x84:0x0](ffff8e8289c163c8), flags 100002 00000002:00100000:1.0:1788201458.065408:0:41142:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8290fe0a80 x1875062187399808/t77309411667(77309411667) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201458.065484:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201458.065553:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe0700 x1875062187400064/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201458.065577:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe0700 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187400064:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201458.065703:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201458.069770:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe0700 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187400064:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201458.069807:0:41142:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8290fe0a80 x1875062187399808/t77309411667(77309411667) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201458.069874:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201458.069898:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201458.069954:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1648 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.069977:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1648 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.069988:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1648 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201458.070180:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe1c00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187400192:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201458.070254:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201458.100398:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288badc00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187401344:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201458.100448:0:41142:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8288baed80 x1875062187401088/t77309411671(77309411671) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201458.100516:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201458.100534:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201458.100622:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1664 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.100632:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1664 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.100642:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1664 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201458.100662:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe2300 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187401472:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201458.101255:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c13c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201458.166693:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721c00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187403648:192.168.201.121@tcp:101:createmany.0 00010000:00010000:0.0:1788201458.166723:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948de00/0x9ae3109cee9e1695 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe7cf expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.166755:0:41142:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948de00/0x9ae3109cee9e1695 lrc: 4/0,1 mode: --/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe7cf expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.166764:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948de00/0x9ae3109cee9e1695 lrc: 4/0,1 mode: --/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe7cf expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.166775:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948de00/0x9ae3109cee9e1695 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe7cf expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.166804:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948de00/0x9ae3109cee9e1695 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe7cf expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.166831:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948de00/0x9ae3109cee9e1695 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe7cf expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.166841:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948de00/0x9ae3109cee9e1695 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe7cf expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.166849:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948de00/0x9ae3109cee9e1695 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe7cf expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201458.166861:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e828d721c00 x1875062187403648/t77309411679(77309411679) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201458.166875:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d721c00 x1875062187403648/t77309411679(77309411679) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201458.166890:0:41142:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948de00/0x9ae3109cee9e1695 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe7cf expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201458.166904:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948de00/0x9ae3109cee9e1695 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe7cf expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.166915:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/138 (0 0) 00000080:00200000:0.0:1788201458.166920:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201458.166933:0:41142:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-1-23, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:0.0:1788201458.166981:0:41142:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0x8a:0x0](ffff8e8289c0d348) 00000080:00200000:0.0:1788201458.167290:0:41142:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x8a:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201458.167296:0:41142:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0x8a:0x0](ffff8e8289c0d348) 00000080:00010000:0.0:1788201458.167299:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x8a:0x0](ffff8e8289c0d348) for lock 0x9ae3109cee9e1695 00000080:00010000:0.0:1788201458.167327:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x8a:0x0](ffff8e8289c0d348) 00000080:00200008:0.0:1788201458.167341:0:41142:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-1-23:[0x200006991:0x8a:0x0](ffff8e8289c0d348), flags 100102 00000002:00100000:0.0:1788201458.167362:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d721c00 x1875062187403648/t77309411679(77309411679) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201458.167394:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1695 from it ffff8e8290e1f1e0 00010000:00010000:0.0:1788201458.167399:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948de00/0x9ae3109cee9e1695 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbe7cf expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.167410:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948de00/0x9ae3109cee9e1695 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbe7cf expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201458.167444:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0x8a:0x0](ffff8e8289c0d348), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201458.167471:0:41142:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-1-23:[0x200006991:0x8a:0x0](ffff8e8289c0d348), flags 100002 00000002:00100000:0.0:1788201458.167521:0:41142:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e828d721c00 x1875062187403648/t77309411679(77309411679) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201458.167532:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201458.167537:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8288b91f80 x1875062187403904/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201458.167557:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b91f80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187403904:192.168.201.121@tcp:35:createmany.0 00000100:00100000:0.0:1788201458.167677:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287be0900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201458.175508:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b91f80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187403904:192.168.201.121@tcp:35:createmany.0 00000002:00100000:0.0:1788201458.175556:0:41142:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e828d721c00 x1875062187403648/t77309411679(77309411679) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201458.175625:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201458.175641:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201458.175704:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e169c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.175716:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e169c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.175726:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e169c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201458.175745:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b90700 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187404032:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201458.175828:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287be0900 going to sleep for 11 seconds 00000100:00100000:1.0:1788201458.180929:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b90700 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187404032:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201458.180970:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e169c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe7e4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.180984:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e169c lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe7e4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.181004:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e169c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe7e4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.181016:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e169c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe7e4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.181028:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e169c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe7e4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.181038:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e169c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe7e4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.181046:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e169c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe7e4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201458.181057:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8288b90700 x1875062187404032/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201458.181072:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e169c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe7e4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.181085:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201458.181092:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201458.181121:0:41142:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 16384 00000080:00010000:1.0:1788201458.181128:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e169c 00000080:00010000:1.0:1788201458.181133:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201458.181137:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e169c from it ffffaa1801fcfbf8 00010000:00010000:1.0:1788201458.181140:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e169c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe7e4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.181150:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e169c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe7e4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201458.181172:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201458.181189:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201458.181205:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201458.181214:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e169c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe7e4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201458.181226:0:41142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e169c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe7e4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201458.181233:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201458.181237:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201458.181240:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e169c 00000080:00010000:1.0:1788201458.181244:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e169c from it ffffaa1801fcfb38 00010000:00010000:1.0:1788201458.181248:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e169c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe7e4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201458.181255:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e169c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe7e4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201458.181275:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201458.181280:0:41142:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-1-24, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8282990600, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201458.181288:0:41142:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-24, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:1.0:1788201458.181336:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-24,[0x200006991:0x8b:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201458.181348:0:41142:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e169c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbe7e4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201458.181394:0:41142:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e169c lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe7e4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201458.181402:0:41142:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201458.181405:0:41142:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e169c lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe7e4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201458.181448:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e16a3 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.181455:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e16a3 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201458.181466:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201458.181473:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe1f80 x1875062187404288/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201458.181483:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e16a3 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201458.181511:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe1f80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187404288:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201458.181632:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201458.188519:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe1f80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187404288:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201458.188548:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e16a3 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe800 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.188576:0:41142:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e16a3 lrc: 4/0,1 mode: --/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe800 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.188595:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e16a3 lrc: 4/0,1 mode: --/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe800 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.188607:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e16a3 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe800 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.188619:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e16a3 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe800 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.188634:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e16a3 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe800 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.188643:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e16a3 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe800 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.188651:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e16a3 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe800 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201458.188663:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8290fe1f80 x1875062187404288/t77309411681(77309411681) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201458.188691:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290fe1f80 x1875062187404288/t77309411681(77309411681) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201458.188707:0:41142:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e16a3 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe800 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201458.188720:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e16a3 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe800 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.188731:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/139 (0 0) 00000080:00200000:1.0:1788201458.188737:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201458.188749:0:41142:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-1-24, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:1.0:1788201458.188778:0:41142:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0x8b:0x0](ffff8e8289c10908) 00000080:00200000:1.0:1788201458.188944:0:41142:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x8b:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201458.188949:0:41142:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0x8b:0x0](ffff8e8289c10908) 00000080:00010000:1.0:1788201458.188953:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x8b:0x0](ffff8e8289c10908) for lock 0x9ae3109cee9e16a3 00000080:00010000:1.0:1788201458.188973:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x8b:0x0](ffff8e8289c10908) 00000080:00200008:1.0:1788201458.188988:0:41142:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-1-24:[0x200006991:0x8b:0x0](ffff8e8289c10908), flags 100102 00000002:00100000:1.0:1788201458.189018:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290fe1f80 x1875062187404288/t77309411681(77309411681) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201458.189050:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e16a3 from it ffff8e82849b94e0 00010000:00010000:1.0:1788201458.189055:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e16a3 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbe800 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.189066:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e16a3 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbe800 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201458.189097:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0x8b:0x0](ffff8e8289c10908), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201458.189124:0:41142:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-1-24:[0x200006991:0x8b:0x0](ffff8e8289c10908), flags 100002 00000002:00100000:1.0:1788201458.189163:0:41142:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8290fe1f80 x1875062187404288/t77309411681(77309411681) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201458.189174:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201458.189180:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe2a00 x1875062187404544/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201458.189200:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe2a00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187404544:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201458.189444:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1d80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201458.195202:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe2a00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187404544:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201458.195238:0:41142:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8290fe1f80 x1875062187404288/t77309411681(77309411681) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201458.195348:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201458.195365:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201458.195602:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992000/0x9ae3109cee9e16aa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.195613:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992000/0x9ae3109cee9e16aa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.195624:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992000/0x9ae3109cee9e16aa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201458.195645:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe2300 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187404672:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201458.195733:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1d80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201458.218208:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288bae680 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187405312:192.168.201.121@tcp:101:createmany.0 00010000:00010000:0.0:1788201458.218234:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f600/0x9ae3109cee9e16b8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe846 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.218248:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f600/0x9ae3109cee9e16b8 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe846 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.218261:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f600/0x9ae3109cee9e16b8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe846 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.218272:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f600/0x9ae3109cee9e16b8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe846 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.218297:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f600/0x9ae3109cee9e16b8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe846 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.218305:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f600/0x9ae3109cee9e16b8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe846 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.218330:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f600/0x9ae3109cee9e16b8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe846 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201458.218342:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8288bae680 x1875062187405312/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201458.218356:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f600/0x9ae3109cee9e16b8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe846 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.218369:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201458.218375:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201458.218401:0:41142:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 16384 00000080:00010000:0.0:1788201458.218407:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e16b8 00000080:00010000:0.0:1788201458.218423:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201458.218427:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e16b8 from it ffffaa1801fcfbf8 00010000:00010000:0.0:1788201458.218431:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f600/0x9ae3109cee9e16b8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe846 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.218440:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f600/0x9ae3109cee9e16b8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe846 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201458.218479:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201458.218503:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201458.218518:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201458.218527:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f600/0x9ae3109cee9e16b8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe846 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201458.218539:0:41142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f600/0x9ae3109cee9e16b8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe846 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201458.218546:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201458.218550:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201458.218553:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e16b8 00000080:00010000:0.0:1788201458.218557:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e16b8 from it ffffaa1801fcfb38 00010000:00010000:0.0:1788201458.218560:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f600/0x9ae3109cee9e16b8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe846 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201458.218569:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f600/0x9ae3109cee9e16b8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe846 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201458.218577:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201458.218582:0:41142:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-1-26, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8287b36780, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201458.218600:0:41142:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-26, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:0.0:1788201458.218618:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-26,[0x200006991:0x8d:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201458.218628:0:41142:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f600/0x9ae3109cee9e16b8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbe846 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.218684:0:41142:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f600/0x9ae3109cee9e16b8 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe846 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.218691:0:41142:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201458.218695:0:41142:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f600/0x9ae3109cee9e16b8 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe846 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.218728:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e16bf lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.218736:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e16bf lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201458.218747:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201458.218753:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8288b92d80 x1875062187405568/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201458.218763:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e16bf lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201458.218782:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b92d80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187405568:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201458.218903:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287be0900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201458.293143:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f1b480 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187407872:192.168.201.121@tcp:101:createmany.0 00010000:00010000:0.0:1788201458.293170:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948cc00/0x9ae3109cee9e16f0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe90a expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.293181:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948cc00/0x9ae3109cee9e16f0 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe90a expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.293203:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948cc00/0x9ae3109cee9e16f0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe90a expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.293214:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948cc00/0x9ae3109cee9e16f0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe90a expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.293228:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948cc00/0x9ae3109cee9e16f0 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe90a expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.293235:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948cc00/0x9ae3109cee9e16f0 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe90a expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.293242:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948cc00/0x9ae3109cee9e16f0 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe90a expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201458.293252:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290f1b480 x1875062187407872/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201458.293265:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948cc00/0x9ae3109cee9e16f0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe90a expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.293278:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201458.293285:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201458.293335:0:41142:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 16384 00000080:00010000:0.0:1788201458.293342:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e16f0 00000080:00010000:0.0:1788201458.293347:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201458.293350:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e16f0 from it ffffaa1801fcfbf8 00010000:00010000:0.0:1788201458.293359:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948cc00/0x9ae3109cee9e16f0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe90a expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.293368:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948cc00/0x9ae3109cee9e16f0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe90a expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201458.293394:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201458.293410:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201458.293553:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201458.293564:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948cc00/0x9ae3109cee9e16f0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe90a expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201458.293575:0:41142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948cc00/0x9ae3109cee9e16f0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe90a expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201458.293581:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201458.293596:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201458.293599:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e16f0 00000080:00010000:0.0:1788201458.293604:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e16f0 from it ffffaa1801fcfb38 00010000:00010000:0.0:1788201458.293607:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948cc00/0x9ae3109cee9e16f0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbe90a expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201458.293614:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948cc00/0x9ae3109cee9e16f0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbe90a expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201458.293623:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201458.293628:0:41142:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-1-30, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8291da5680, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201458.293637:0:41142:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-30, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:0.0:1788201458.293661:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-30,[0x200006991:0x91:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201458.293673:0:41142:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948cc00/0x9ae3109cee9e16f0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbe90a expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.293727:0:41142:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948cc00/0x9ae3109cee9e16f0 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe90a expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.293735:0:41142:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201458.293738:0:41142:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948cc00/0x9ae3109cee9e16f0 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbe90a expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.293773:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e16f7 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.293780:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e16f7 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201458.293793:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201458.293800:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8288b90000 x1875062187408128/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201458.294156:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e16f7 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201458.294181:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b90000 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187408128:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201458.296993:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287be0900 going to sleep for 11 seconds 00000002:00100000:1.0:1788201458.313794:0:41142:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8288b90000 x1875062187408128/t77309411693(77309411693) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201458.313874:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201458.313892:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201458.313948:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993600/0x9ae3109cee9e16fe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.313958:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993600/0x9ae3109cee9e16fe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.313968:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993600/0x9ae3109cee9e16fe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201458.313988:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe1c00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187408512:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201458.314060:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1d80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201458.420936:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f18700 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187412224:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201458.420972:0:41142:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8290f19c00 x1875062187411968/t77309411705(77309411705) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201458.421053:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201458.421071:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201458.421123:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1752 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.421133:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1752 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.421141:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1752 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201458.421169:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe0000 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187412352:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201458.421237:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c13c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201458.423747:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe0000 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187412352:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201458.423774:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1752 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbea61 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.423788:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1752 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbea61 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.423799:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1752 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbea61 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.423882:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1752 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbea61 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.423897:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1752 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbea61 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.423905:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1752 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbea61 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.423912:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1752 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbea61 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201458.423924:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290fe0000 x1875062187412352/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201458.423957:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1752 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbea61 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.423969:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201458.423976:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201458.424104:0:41142:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 16384 00000080:00010000:1.0:1788201458.424112:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1752 00000080:00010000:1.0:1788201458.424118:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201458.424121:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1752 from it ffffaa1801fcfbf8 00010000:00010000:1.0:1788201458.424125:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1752 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbea61 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.424134:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1752 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbea61 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201458.424158:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201458.424180:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201458.424195:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201458.424204:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1752 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbea61 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201458.424214:0:41142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1752 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbea61 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201458.424221:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201458.424225:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201458.424228:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1752 00000080:00010000:1.0:1788201458.424240:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1752 from it ffffaa1801fcfb38 00010000:00010000:1.0:1788201458.424244:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1752 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbea61 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201458.424252:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1752 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbea61 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201458.424261:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201458.424267:0:41142:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-1-37, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e828d01a900, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201458.424275:0:41142:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-37, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:1.0:1788201458.424294:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-37,[0x200006991:0x98:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201458.424336:0:41142:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1752 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbea61 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201458.424385:0:41142:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1752 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbea61 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201458.424392:0:41142:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201458.424395:0:41142:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1752 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbea61 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201458.424607:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1759 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.424615:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1759 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201458.424626:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201458.424634:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe0000 x1875062187412608/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201458.424650:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1759 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201458.424669:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe0000 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187412608:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201458.424792:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c13c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201458.477817:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187413888:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201458.477845:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f200/0x9ae3109cee9e1775 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeadf expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.477878:0:41142:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f200/0x9ae3109cee9e1775 lrc: 4/0,1 mode: --/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeadf expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.477887:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f200/0x9ae3109cee9e1775 lrc: 4/0,1 mode: --/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeadf expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.477899:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f200/0x9ae3109cee9e1775 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeadf expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.477910:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f200/0x9ae3109cee9e1775 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeadf expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.477924:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f200/0x9ae3109cee9e1775 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeadf expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.477933:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f200/0x9ae3109cee9e1775 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeadf expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.477958:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f200/0x9ae3109cee9e1775 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeadf expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201458.477972:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e82894e4e00 x1875062187413888/t77309411711(77309411711) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201458.477985:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e82894e4e00 x1875062187413888/t77309411711(77309411711) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201458.478014:0:41142:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f200/0x9ae3109cee9e1775 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeadf expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201458.478028:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f200/0x9ae3109cee9e1775 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeadf expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.478039:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/154 (0 0) 00000080:00200000:1.0:1788201458.478045:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201458.478059:0:41142:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-1-39, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:1.0:1788201458.478087:0:41142:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0x9a:0x0](ffff8e8289c14b08) 00000080:00200000:1.0:1788201458.478219:0:41142:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x9a:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201458.478225:0:41142:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0x9a:0x0](ffff8e8289c14b08) 00000080:00010000:1.0:1788201458.478228:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x9a:0x0](ffff8e8289c14b08) for lock 0x9ae3109cee9e1775 00000080:00010000:1.0:1788201458.478238:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x9a:0x0](ffff8e8289c14b08) 00000080:00200008:1.0:1788201458.478251:0:41142:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-1-39:[0x200006991:0x9a:0x0](ffff8e8289c14b08), flags 100102 00000002:00100000:1.0:1788201458.478270:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e82894e4e00 x1875062187413888/t77309411711(77309411711) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201458.478337:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1775 from it ffff8e828d664540 00010000:00010000:1.0:1788201458.478343:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f200/0x9ae3109cee9e1775 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbeadf expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.478353:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f200/0x9ae3109cee9e1775 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbeadf expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201458.478373:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0x9a:0x0](ffff8e8289c14b08), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201458.478399:0:41142:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-1-39:[0x200006991:0x9a:0x0](ffff8e8289c14b08), flags 100002 00000002:00100000:1.0:1788201458.478440:0:41142:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e82894e4e00 x1875062187413888/t77309411711(77309411711) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201458.478450:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201458.478456:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe1880 x1875062187414144/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201458.478476:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe1880 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187414144:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201458.478646:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c16c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201458.487571:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f1bb80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187414272:192.168.201.121@tcp:101:createmany.0 00010000:00010000:0.0:1788201458.487633:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d400/0x9ae3109cee9e177c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeaf4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.487666:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d400/0x9ae3109cee9e177c lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeaf4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.487677:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d400/0x9ae3109cee9e177c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeaf4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.487688:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d400/0x9ae3109cee9e177c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeaf4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.487701:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d400/0x9ae3109cee9e177c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeaf4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.487708:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d400/0x9ae3109cee9e177c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeaf4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.487715:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d400/0x9ae3109cee9e177c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeaf4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201458.487725:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290f1bb80 x1875062187414272/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201458.487738:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d400/0x9ae3109cee9e177c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeaf4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.487750:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201458.487758:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201458.487786:0:41142:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 16384 00000080:00010000:0.0:1788201458.487797:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e177c 00000080:00010000:0.0:1788201458.487803:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201458.487806:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e177c from it ffffaa1801fcfbf8 00010000:00010000:0.0:1788201458.487823:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d400/0x9ae3109cee9e177c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeaf4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.487832:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d400/0x9ae3109cee9e177c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeaf4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201458.487857:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201458.487882:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201458.487897:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201458.487906:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d400/0x9ae3109cee9e177c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeaf4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201458.487915:0:41142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d400/0x9ae3109cee9e177c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeaf4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201458.487921:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201458.487925:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201458.487928:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e177c 00000080:00010000:0.0:1788201458.487931:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e177c from it ffffaa1801fcfb38 00010000:00010000:0.0:1788201458.487933:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d400/0x9ae3109cee9e177c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeaf4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201458.487940:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d400/0x9ae3109cee9e177c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeaf4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201458.487960:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201458.487965:0:41142:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-1-40, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8291da5500, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201458.487973:0:41142:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-40, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:0.0:1788201458.487993:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-40,[0x200006991:0x9b:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201458.488015:0:41142:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d400/0x9ae3109cee9e177c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbeaf4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.488069:0:41142:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d400/0x9ae3109cee9e177c lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbeaf4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.488076:0:41142:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201458.488079:0:41142:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d400/0x9ae3109cee9e177c lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbeaf4 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.488112:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e1783 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.488120:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e1783 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201458.488131:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201458.488138:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8288b90380 x1875062187414528/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201458.488148:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e1783 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201458.488381:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b90380 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187414528:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201458.488500:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287be0900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201458.494660:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b90380 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187414528:192.168.201.121@tcp:101:createmany.0 00010000:00010000:0.0:1788201458.494690:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e1783 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb10 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.494720:0:41142:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e1783 lrc: 4/0,1 mode: --/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb10 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.494729:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e1783 lrc: 4/0,1 mode: --/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb10 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.494740:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e1783 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeb10 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.494751:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e1783 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb10 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.494764:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e1783 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeb10 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.494773:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e1783 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeb10 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.494781:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e1783 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeb10 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201458.494793:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8288b90380 x1875062187414528/t77309411713(77309411713) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201458.494831:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8288b90380 x1875062187414528/t77309411713(77309411713) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201458.494846:0:41142:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e1783 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb10 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201458.494859:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e1783 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb10 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.494871:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/155 (0 0) 00000080:00200000:0.0:1788201458.494878:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201458.494891:0:41142:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-1-40, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:0.0:1788201458.494920:0:41142:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0x9b:0x0](ffff8e8289c0db88) 00000080:00200000:0.0:1788201458.494986:0:41142:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x9b:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201458.494990:0:41142:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0x9b:0x0](ffff8e8289c0db88) 00000080:00010000:0.0:1788201458.494994:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x9b:0x0](ffff8e8289c0db88) for lock 0x9ae3109cee9e1783 00000080:00010000:0.0:1788201458.495014:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x9b:0x0](ffff8e8289c0db88) 00000080:00200008:0.0:1788201458.495028:0:41142:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-1-40:[0x200006991:0x9b:0x0](ffff8e8289c0db88), flags 100102 00000002:00100000:0.0:1788201458.495050:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8288b90380 x1875062187414528/t77309411713(77309411713) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201458.495082:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1783 from it ffff8e82855fc5a0 00010000:00010000:0.0:1788201458.495087:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e1783 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbeb10 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.495096:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e1783 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbeb10 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201458.495127:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0x9b:0x0](ffff8e8289c0db88), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201458.495165:0:41142:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-1-40:[0x200006991:0x9b:0x0](ffff8e8289c0db88), flags 100002 00000002:00100000:0.0:1788201458.495206:0:41142:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8288b90380 x1875062187414528/t77309411713(77309411713) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201458.495216:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201458.495221:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8288b90700 x1875062187414784/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201458.495240:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b90700 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187414784:192.168.201.121@tcp:35:createmany.0 00000100:00100000:0.0:1788201458.495341:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82910d20c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201458.500651:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b90700 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187414784:192.168.201.121@tcp:35:createmany.0 00000002:00100000:0.0:1788201458.500690:0:41142:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8288b90380 x1875062187414528/t77309411713(77309411713) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201458.500754:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201458.500770:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201458.500837:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e178a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.500847:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e178a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.500857:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e178a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201458.500876:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b90a80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187414912:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201458.500968:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82910d20c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201458.504237:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b90a80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187414912:192.168.201.121@tcp:101:createmany.0 00010000:00010000:0.0:1788201458.504262:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e178a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb25 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.504274:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e178a lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb25 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.504286:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e178a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeb25 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.504297:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e178a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb25 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.504325:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e178a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeb25 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.504334:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e178a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeb25 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.504341:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e178a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeb25 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201458.504352:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8288b90a80 x1875062187414912/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201458.504365:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e178a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb25 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.504395:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201458.504401:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201458.504480:0:41142:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 16384 00000080:00010000:0.0:1788201458.504538:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e178a 00000080:00010000:0.0:1788201458.504543:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201458.504547:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e178a from it ffffaa1801fcfbf8 00010000:00010000:0.0:1788201458.504550:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e178a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeb25 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.504559:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e178a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb25 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201458.504590:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201458.504608:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201458.504621:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201458.504629:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e178a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb25 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201458.504639:0:41142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e178a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb25 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201458.504645:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201458.504649:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201458.504651:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e178a 00000080:00010000:0.0:1788201458.504655:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e178a from it ffffaa1801fcfb38 00010000:00010000:0.0:1788201458.504658:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e178a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeb25 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201458.504670:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e178a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb25 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201458.504678:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201458.504682:0:41142:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-1-41, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8282990f00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201458.504689:0:41142:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-41, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:0.0:1788201458.504705:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-41,[0x200006991:0x9c:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201458.504716:0:41142:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e178a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbeb25 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.504760:0:41142:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e178a lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbeb25 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.504767:0:41142:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201458.504770:0:41142:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e178a lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbeb25 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.504799:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1791 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.504869:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1791 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201458.504881:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201458.504888:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8288b90a80 x1875062187415168/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201458.504897:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1791 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201458.504921:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b90a80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187415168:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201458.505243:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82910d20c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201458.525612:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290f19500 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187415424:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201458.525649:0:41142:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8288b90a80 x1875062187415168/t77309411715(77309411715) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201458.525720:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201458.525740:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201458.525802:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1798 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.525949:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1798 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.525960:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1798 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201458.525981:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe3480 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187415552:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201458.526128:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1600 going to sleep for 11 seconds 00000100:00100000:1.0:1788201458.529474:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe3480 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187415552:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201458.529510:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1798 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb56 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.529523:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1798 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb56 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.529535:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1798 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeb56 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.529546:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1798 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb56 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.529558:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1798 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeb56 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.529567:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1798 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeb56 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.529600:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1798 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeb56 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201458.529613:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290fe3480 x1875062187415552/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201458.529629:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1798 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb56 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.529642:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201458.529650:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201458.529674:0:41142:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 16384 00000080:00010000:1.0:1788201458.529682:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1798 00000080:00010000:1.0:1788201458.529687:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201458.529690:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1798 from it ffffaa1801fcfbf8 00010000:00010000:1.0:1788201458.529693:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1798 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeb56 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.529703:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1798 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb56 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201458.529727:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201458.529747:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201458.529761:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201458.529770:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1798 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb56 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201458.529792:0:41142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1798 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb56 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201458.529799:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201458.529804:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201458.529817:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1798 00000080:00010000:1.0:1788201458.529822:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1798 from it ffffaa1801fcfb38 00010000:00010000:1.0:1788201458.529825:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1798 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbeb56 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201458.529833:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1798 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbeb56 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201458.529842:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201458.529848:0:41142:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-1-42, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8284c98d80, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201458.529857:0:41142:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-42, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:1.0:1788201458.529877:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-42,[0x200006991:0x9d:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201458.529888:0:41142:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1798 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbeb56 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201458.529934:0:41142:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1798 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbeb56 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201458.529941:0:41142:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201458.529944:0:41142:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992200/0x9ae3109cee9e1798 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbeb56 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201458.529990:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e179f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.529998:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e179f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201458.530008:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201458.530015:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe3480 x1875062187415808/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201458.530025:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e179f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201458.530044:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe3480 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187415808:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201458.530171:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201458.625192:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187419008:192.168.201.121@tcp:101:createmany.0 00010000:00010000:0.0:1788201458.625222:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948fe00/0x9ae3109cee9e17e5 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbec67 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.625255:0:41142:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948fe00/0x9ae3109cee9e17e5 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xa2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbec67 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.625262:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948fe00/0x9ae3109cee9e17e5 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xa2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbec67 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.625273:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948fe00/0x9ae3109cee9e17e5 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xa2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbec67 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.625285:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948fe00/0x9ae3109cee9e17e5 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xa2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbec67 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.625297:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948fe00/0x9ae3109cee9e17e5 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xa2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbec67 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.625306:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948fe00/0x9ae3109cee9e17e5 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xa2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbec67 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.625327:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948fe00/0x9ae3109cee9e17e5 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xa2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbec67 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201458.625360:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e82894e5f80 x1875062187419008/t77309411727(77309411727) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201458.625373:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e82894e5f80 x1875062187419008/t77309411727(77309411727) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201458.625388:0:41142:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948fe00/0x9ae3109cee9e17e5 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xa2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbec67 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201458.625402:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948fe00/0x9ae3109cee9e17e5 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xa2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbec67 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.625631:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/162 (0 0) 00000080:00200000:0.0:1788201458.625638:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201458.625653:0:41142:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-1-47, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:0.0:1788201458.625680:0:41142:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xa2:0x0](ffff8e8289c09988) 00000080:00200000:0.0:1788201458.625751:0:41142:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xa2:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201458.625756:0:41142:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xa2:0x0](ffff8e8289c09988) 00000080:00010000:0.0:1788201458.625760:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xa2:0x0](ffff8e8289c09988) for lock 0x9ae3109cee9e17e5 00000080:00010000:0.0:1788201458.625769:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xa2:0x0](ffff8e8289c09988) 00000080:00200008:0.0:1788201458.625782:0:41142:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-1-47:[0x200006991:0xa2:0x0](ffff8e8289c09988), flags 100102 00000002:00100000:0.0:1788201458.625802:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e82894e5f80 x1875062187419008/t77309411727(77309411727) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201458.625904:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e17e5 from it ffff8e828d664ea0 00010000:00010000:0.0:1788201458.625909:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948fe00/0x9ae3109cee9e17e5 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xa2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbec67 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.625930:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948fe00/0x9ae3109cee9e17e5 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xa2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbec67 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201458.625950:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xa2:0x0](ffff8e8289c09988), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201458.625977:0:41142:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-1-47:[0x200006991:0xa2:0x0](ffff8e8289c09988), flags 100002 00000002:00100000:0.0:1788201458.626021:0:41142:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e82894e5f80 x1875062187419008/t77309411727(77309411727) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201458.626032:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201458.626037:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8288b93100 x1875062187419264/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201458.626056:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b93100 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187419264:192.168.201.121@tcp:35:createmany.0 00000100:00100000:0.0:1788201458.626720:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287e50cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201458.663495:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64ac00 going to sleep for 11 seconds 00000100:00100000:0.0:1788201458.673553:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187420544:192.168.201.121@tcp:35:createmany.0 00000002:00100000:0.0:1788201458.673601:0:41142:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e82894e6d80 x1875062187420288/t77309411731(77309411731) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201458.673676:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201458.673695:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201458.673765:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1808 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.673775:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1808 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.673785:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1808 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201458.673827:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b93480 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187420672:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201458.673893:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287e50180 going to sleep for 11 seconds 00000100:00100000:0.0:1788201458.679562:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b93480 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187420672:192.168.201.121@tcp:101:createmany.0 00010000:00010000:0.0:1788201458.679599:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1808 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbecde expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.679612:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1808 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbecde expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.679625:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1808 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbecde expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.679636:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1808 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbecde expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.679650:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1808 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbecde expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.679659:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1808 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbecde expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.679667:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1808 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbecde expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201458.679680:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8288b93480 x1875062187420672/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201458.679716:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1808 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbecde expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.679731:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201458.679738:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201458.679764:0:41142:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 16384 00000080:00010000:0.0:1788201458.679769:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1808 00000080:00010000:0.0:1788201458.679775:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201458.679778:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1808 from it ffffaa1801fcfbf8 00010000:00010000:0.0:1788201458.679781:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1808 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbecde expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.679791:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1808 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbecde expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201458.679826:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201458.679845:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201458.679859:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201458.679868:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1808 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbecde expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201458.679879:0:41142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1808 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbecde expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201458.679886:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201458.679891:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201458.679894:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1808 00000080:00010000:0.0:1788201458.679902:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1808 from it ffffaa1801fcfb38 00010000:00010000:0.0:1788201458.679906:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1808 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbecde expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201458.679914:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1808 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbecde expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201458.679923:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201458.679928:0:41142:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-1-50, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8283ce6900, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201458.679937:0:41142:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-50, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:0.0:1788201458.679953:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-50,[0x200006991:0xa5:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201458.679964:0:41142:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1808 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbecde expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.680378:0:41142:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1808 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbecde expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.680386:0:41142:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201458.680389:0:41142:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1808 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbecde expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.680437:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e180f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.680445:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e180f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201458.680456:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201458.680465:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8288b93480 x1875062187420928/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201458.680634:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e180f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201458.680724:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b93480 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187420928:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201458.680868:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201458.691996:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187421184:192.168.201.121@tcp:35:createmany.0 00000002:00100000:0.0:1788201458.692035:0:41142:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8288b93480 x1875062187420928/t77309411733(77309411733) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201458.692116:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201458.692136:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201458.692198:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1816 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.692208:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1816 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.692219:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1816 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201458.692240:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b92300 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187421312:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201458.692633:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287e50180 going to sleep for 11 seconds 00000100:00100000:0.0:1788201458.715076:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5500 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187421952:192.168.201.121@tcp:101:createmany.0 00010000:00010000:0.0:1788201458.715106:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1824 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbed40 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.715119:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1824 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbed40 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.715132:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1824 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbed40 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.715154:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1824 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbed40 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.715167:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1824 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbed40 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.715175:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1824 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbed40 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.715182:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1824 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbed40 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201458.715194:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894e5500 x1875062187421952/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201458.715208:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1824 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbed40 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.715222:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201458.715230:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201458.715258:0:41142:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 16384 00000080:00010000:0.0:1788201458.715264:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1824 00000080:00010000:0.0:1788201458.715270:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201458.715273:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1824 from it ffffaa1801fcfbf8 00010000:00010000:0.0:1788201458.715276:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1824 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbed40 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.715296:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1824 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbed40 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201458.715349:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201458.715369:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201458.715385:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201458.715395:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1824 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbed40 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201458.715406:0:41142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1824 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbed40 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201458.715414:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201458.715418:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201458.715421:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1824 00000080:00010000:0.0:1788201458.715424:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1824 from it ffffaa1801fcfb38 00010000:00010000:0.0:1788201458.715427:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1824 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbed40 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201458.715434:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1824 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbed40 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201458.715442:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201458.715447:0:41142:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-1-52, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8287b36900, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201458.715456:0:41142:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-52, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:0.0:1788201458.715474:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-52,[0x200006991:0xa7:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201458.715502:0:41142:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1824 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbed40 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.715552:0:41142:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1824 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbed40 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.715560:0:41142:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201458.715563:0:41142:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1824 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbed40 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.715607:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e182b lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.715614:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e182b lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201458.715624:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201458.715630:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8288b90e00 x1875062187422208/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201458.715640:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e182b lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201458.715658:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b90e00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187422208:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201458.715779:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287e50b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201458.720297:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b90e00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187422208:192.168.201.121@tcp:101:createmany.0 00010000:00010000:0.0:1788201458.720360:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e182b lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbed5c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.720389:0:41142:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e182b lrc: 4/0,1 mode: --/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbed5c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.720414:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e182b lrc: 4/0,1 mode: --/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbed5c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.720425:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e182b lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbed5c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.720436:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e182b lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbed5c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.720453:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e182b lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbed5c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.720461:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e182b lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbed5c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.720469:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e182b lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbed5c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201458.720492:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8288b90e00 x1875062187422208/t77309411737(77309411737) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201458.720506:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8288b90e00 x1875062187422208/t77309411737(77309411737) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201458.720520:0:41142:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e182b lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbed5c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201458.720535:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e182b lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbed5c expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.720553:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/167 (0 0) 00000080:00200000:0.0:1788201458.720559:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201458.720574:0:41142:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-1-52, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:0.0:1788201458.720619:0:41142:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xa7:0x0](ffff8e8289c0cb08) 00000080:00200000:0.0:1788201458.720702:0:41142:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xa7:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201458.720708:0:41142:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xa7:0x0](ffff8e8289c0cb08) 00000080:00010000:0.0:1788201458.720711:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xa7:0x0](ffff8e8289c0cb08) for lock 0x9ae3109cee9e182b 00000080:00010000:0.0:1788201458.720723:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xa7:0x0](ffff8e8289c0cb08) 00000080:00200008:0.0:1788201458.720737:0:41142:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-1-52:[0x200006991:0xa7:0x0](ffff8e8289c0cb08), flags 100102 00000002:00100000:0.0:1788201458.720759:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8288b90e00 x1875062187422208/t77309411737(77309411737) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201458.720793:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e182b from it ffff8e82855fcea0 00010000:00010000:0.0:1788201458.720798:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e182b lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbed5c expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.720820:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e182b lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbed5c expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201458.720844:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xa7:0x0](ffff8e8289c0cb08), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201458.720873:0:41142:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-1-52:[0x200006991:0xa7:0x0](ffff8e8289c0cb08), flags 100002 00000002:00100000:0.0:1788201458.720922:0:41142:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8288b90e00 x1875062187422208/t77309411737(77309411737) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201458.720934:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201458.720940:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8288b91500 x1875062187422464/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201458.720975:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b91500 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187422464:192.168.201.121@tcp:35:createmany.0 00000100:00100000:0.0:1788201458.721202:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287e500c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201458.771938:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a5500 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187424384:192.168.201.121@tcp:35:createmany.0 00000002:00100000:0.0:1788201458.771982:0:41142:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e828d6a7480 x1875062187424128/t77309411743(77309411743) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201458.772140:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201458.772159:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201458.772218:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e185c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.772228:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e185c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.772239:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e185c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201458.772260:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b92680 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187424512:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201458.772358:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287e50840 going to sleep for 11 seconds 00000080:00200000:1.0:1788201458.915756:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201458.915775:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201458.915971:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992a00/0x9ae3109cee9e18cc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.915981:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992a00/0x9ae3109cee9e18cc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.915991:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992a00/0x9ae3109cee9e18cc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201458.916041:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe3100 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187429632:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201458.916227:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1300 going to sleep for 11 seconds 00000080:00200000:0.0:1788201458.938758:0:41142:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 16384 00000080:00010000:0.0:1788201458.938770:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e18da 00000080:00010000:0.0:1788201458.938776:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201458.938780:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e18da from it ffffaa1801fcfbf8 00010000:00010000:0.0:1788201458.938784:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e18da lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbefbd expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.938795:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e18da lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbefbd expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201458.938981:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201458.939301:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201458.939390:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201458.939590:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e18da lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbefbd expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201458.939604:0:41142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e18da lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbefbd expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201458.939613:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201458.939619:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201458.939622:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e18da 00000080:00010000:0.0:1788201458.939628:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e18da from it ffffaa1801fcfb38 00010000:00010000:0.0:1788201458.939631:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e18da lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbefbd expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201458.939638:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e18da lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbefbd expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201458.939648:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201458.939653:0:41142:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-1-65, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8287b36480, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201458.939661:0:41142:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-65, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:0.0:1788201458.939680:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-65,[0x200006991:0xb4:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201458.939694:0:41142:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e18da lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbefbd expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.940214:0:41142:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e18da lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbefbd expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.940224:0:41142:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201458.940228:0:41142:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e18da lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbefbd expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201458.940269:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e18e1 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201458.940277:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e18e1 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201458.940288:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201458.940299:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8288b91880 x1875062187430528/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201458.940604:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e18e1 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201458.940628:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b91880 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187430528:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201458.941117:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287e50840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201458.964037:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a4e00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187430784:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201458.964072:0:41142:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8288b91880 x1875062187430528/t77309411763(77309411763) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201458.964150:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201458.964168:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201458.964225:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e18e8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.964235:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e18e8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201458.964245:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e18e8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201458.964264:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe0e00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187430912:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201458.964357:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201459.174447:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289553800 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187438208:192.168.201.121@tcp:101:createmany.0 00010000:00010000:0.0:1788201459.174476:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e1989 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf225 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.174519:0:41142:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e1989 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf225 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.174548:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e1989 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf225 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.174559:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e1989 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf225 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.174579:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e1989 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf225 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.174594:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e1989 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf225 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.174602:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e1989 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf225 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.174609:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e1989 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf225 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201459.174621:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8289553800 x1875062187438208/t77309411787(77309411787) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201475 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201459.174634:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289553800 x1875062187438208/t77309411787(77309411787) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201475 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201459.174648:0:41142:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e1989 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf225 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201459.174662:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e1989 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf225 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.174677:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/192 (0 0) 00000080:00200000:0.0:1788201459.174684:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201459.174697:0:41142:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-1-77, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:0.0:1788201459.174729:0:41142:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xc0:0x0](ffff8e8289c09148) 00000080:00200000:0.0:1788201459.174955:0:41142:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xc0:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201459.174962:0:41142:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xc0:0x0](ffff8e8289c09148) 00000080:00010000:0.0:1788201459.174965:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xc0:0x0](ffff8e8289c09148) for lock 0x9ae3109cee9e1989 00000080:00010000:0.0:1788201459.174975:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xc0:0x0](ffff8e8289c09148) 00000080:00200008:0.0:1788201459.174987:0:41142:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-1-77:[0x200006991:0xc0:0x0](ffff8e8289c09148), flags 100102 00000002:00100000:0.0:1788201459.175008:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289553800 x1875062187438208/t77309411787(77309411787) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201475 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201459.175042:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1989 from it ffff8e8289485d80 00010000:00010000:0.0:1788201459.175047:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e1989 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbf225 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.175056:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e1989 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbf225 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201459.175076:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xc0:0x0](ffff8e8289c09148), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201459.175103:0:41142:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-1-77:[0x200006991:0xc0:0x0](ffff8e8289c09148), flags 100002 00000002:00100000:0.0:1788201459.175145:0:41142:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8289553800 x1875062187438208/t77309411787(77309411787) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201475 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201459.175155:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201459.175160:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8288b93800 x1875062187438464/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201459.175192:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b93800 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187438464:192.168.201.121@tcp:35:createmany.0 00000100:00100000:0.0:1788201459.175281:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287e50180 going to sleep for 11 seconds 00000100:00100000:0.0:1788201459.199431:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d703800 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187438848:192.168.201.121@tcp:101:createmany.0 00010000:00010000:0.0:1788201459.199460:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1997 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf256 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.199489:0:41142:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1997 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf256 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.199498:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1997 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf256 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.199724:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1997 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf256 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.199738:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1997 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf256 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.199752:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1997 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf256 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.199760:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1997 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf256 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.199768:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1997 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf256 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201459.199809:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e828d703800 x1875062187438848/t77309411789(77309411789) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201475 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201459.199822:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d703800 x1875062187438848/t77309411789(77309411789) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201475 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201459.199837:0:41142:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1997 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf256 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201459.199851:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1997 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf256 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.199861:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/193 (0 0) 00000080:00200000:0.0:1788201459.199868:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201459.199880:0:41142:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-1-78, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:0.0:1788201459.199906:0:41142:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xc1:0x0](ffff8e8289c0b248) 00000080:00200000:0.0:1788201459.200100:0:41142:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xc1:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201459.200105:0:41142:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xc1:0x0](ffff8e8289c0b248) 00000080:00010000:0.0:1788201459.200108:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xc1:0x0](ffff8e8289c0b248) for lock 0x9ae3109cee9e1997 00000080:00010000:0.0:1788201459.200117:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xc1:0x0](ffff8e8289c0b248) 00000080:00200008:0.0:1788201459.200130:0:41142:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-1-78:[0x200006991:0xc1:0x0](ffff8e8289c0b248), flags 100102 00000002:00100000:0.0:1788201459.200150:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d703800 x1875062187438848/t77309411789(77309411789) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201475 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201459.200182:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1997 from it ffff8e8289485ae0 00010000:00010000:0.0:1788201459.200187:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1997 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbf256 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.200206:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1997 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbf256 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201459.200227:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xc1:0x0](ffff8e8289c0b248), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201459.200255:0:41142:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-1-78:[0x200006991:0xc1:0x0](ffff8e8289c0b248), flags 100002 00000002:00100000:0.0:1788201459.200301:0:41142:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e828d703800 x1875062187438848/t77309411789(77309411789) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201475 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201459.200350:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201459.200356:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8288b91180 x1875062187439104/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201459.200377:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b91180 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187439104:192.168.201.121@tcp:35:createmany.0 00000100:00100000:0.0:1788201459.200460:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287e50180 going to sleep for 11 seconds 00000100:00100000:0.0:1788201459.206342:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b91180 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187439104:192.168.201.121@tcp:35:createmany.0 00000002:00100000:0.0:1788201459.206378:0:41142:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e828d703800 x1875062187438848/t77309411789(77309411789) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201475 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788201459.206439:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201459.206457:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201459.206531:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e199e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.206543:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e199e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.206554:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e199e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201459.206599:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b91c00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187439232:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201459.206665:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287e50180 going to sleep for 11 seconds 00000100:00100000:1.0:1788201459.251209:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d701180 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187439744:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201459.251260:0:41142:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e828d700e00 x1875062187439488/t77309411791(77309411791) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201475 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201459.251420:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201459.251439:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201459.251506:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e19ac lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.251541:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e19ac lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.251553:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e19ac lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201459.251585:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe0e00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187439872:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201459.251780:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1000 going to sleep for 11 seconds 00000100:00100000:1.0:1788201459.255415:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe0e00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187439872:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201459.255442:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e19ac lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf29c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.255455:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e19ac lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf29c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.255467:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e19ac lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf29c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.255478:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e19ac lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf29c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.255490:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e19ac lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf29c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.255498:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e19ac lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf29c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.255538:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e19ac lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf29c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201459.255552:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8290fe0e00 x1875062187439872/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201475 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201459.255566:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e19ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf29c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.255589:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201459.255597:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201459.255621:0:41142:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 16384 00000080:00010000:1.0:1788201459.255627:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e19ac 00000080:00010000:1.0:1788201459.255632:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201459.255636:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e19ac from it ffffaa1801fcfbf8 00010000:00010000:1.0:1788201459.255639:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e19ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf29c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.255649:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e19ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf29c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201459.255672:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201459.255695:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201459.255709:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201459.255718:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e19ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf29c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201459.255739:0:41142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e19ac lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf29c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201459.255747:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201459.255751:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201459.255754:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e19ac 00000080:00010000:1.0:1788201459.255758:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e19ac from it ffffaa1801fcfb38 00010000:00010000:1.0:1788201459.255762:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e19ac lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf29c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201459.255769:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e19ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf29c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201459.255778:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201459.255783:0:41142:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-1-80, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8284c98600, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201459.255791:0:41142:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-80, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:1.0:1788201459.255884:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-80,[0x200006991:0xc3:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201459.255896:0:41142:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e19ac lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbf29c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201459.256017:0:41142:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e19ac lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbf29c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201459.256025:0:41142:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201459.256028:0:41142:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e19ac lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbf29c expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201459.256072:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992a00/0x9ae3109cee9e19b3 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.256079:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992a00/0x9ae3109cee9e19b3 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201459.256090:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201459.256097:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe0e00 x1875062187440128/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201459.256104:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992a00/0x9ae3109cee9e19b3 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201459.256123:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe0e00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187440128:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201459.256243:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1000 going to sleep for 11 seconds 00000100:00100000:1.0:1788201459.260427:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe0e00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187440128:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201459.260455:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992a00/0x9ae3109cee9e19b3 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf2b8 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.260482:0:41142:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992a00/0x9ae3109cee9e19b3 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xc3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf2b8 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.260491:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992a00/0x9ae3109cee9e19b3 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf2b8 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.260501:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992a00/0x9ae3109cee9e19b3 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf2b8 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.260535:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992a00/0x9ae3109cee9e19b3 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf2b8 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.260549:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992a00/0x9ae3109cee9e19b3 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf2b8 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.260557:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992a00/0x9ae3109cee9e19b3 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf2b8 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.260564:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992a00/0x9ae3109cee9e19b3 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf2b8 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201459.260586:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8290fe0e00 x1875062187440128/t77309411793(77309411793) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201475 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201459.260598:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290fe0e00 x1875062187440128/t77309411793(77309411793) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201475 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201459.260610:0:41142:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992a00/0x9ae3109cee9e19b3 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf2b8 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201459.260622:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992a00/0x9ae3109cee9e19b3 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf2b8 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.260632:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/195 (0 0) 00000080:00200000:1.0:1788201459.260638:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201459.260650:0:41142:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-1-80, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:1.0:1788201459.260686:0:41142:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xc3:0x0](ffff8e8289c15b88) 00000080:00200000:1.0:1788201459.260756:0:41142:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xc3:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201459.260762:0:41142:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xc3:0x0](ffff8e8289c15b88) 00000080:00010000:1.0:1788201459.260765:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xc3:0x0](ffff8e8289c15b88) for lock 0x9ae3109cee9e19b3 00000080:00010000:1.0:1788201459.260775:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xc3:0x0](ffff8e8289c15b88) 00000080:00200008:1.0:1788201459.260788:0:41142:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-1-80:[0x200006991:0xc3:0x0](ffff8e8289c15b88), flags 100102 00000002:00100000:1.0:1788201459.260819:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8290fe0e00 x1875062187440128/t77309411793(77309411793) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201475 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201459.260849:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e19b3 from it ffff8e82849b9ba0 00010000:00010000:1.0:1788201459.260854:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992a00/0x9ae3109cee9e19b3 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbf2b8 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.260863:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992a00/0x9ae3109cee9e19b3 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbf2b8 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201459.260883:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xc3:0x0](ffff8e8289c15b88), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201459.260906:0:41142:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-1-80:[0x200006991:0xc3:0x0](ffff8e8289c15b88), flags 100002 00000002:00100000:1.0:1788201459.261130:0:41142:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8290fe0e00 x1875062187440128/t77309411793(77309411793) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201475 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201459.261140:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201459.261146:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8290fe3100 x1875062187440384/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201459.261168:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe3100 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187440384:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201459.261242:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1300 going to sleep for 11 seconds 00000100:00100000:0.0:1788201459.398842:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d701f80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187443072:192.168.201.121@tcp:101:createmany.0 00010000:00010000:0.0:1788201459.398868:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e19f2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf391 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.398881:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e19f2 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf391 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.398892:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e19f2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf391 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.398903:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e19f2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf391 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.398914:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e19f2 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf391 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.398922:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e19f2 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf391 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.398929:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e19f2 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf391 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201459.398941:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d701f80 x1875062187443072/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201475 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201459.398980:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e19f2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf391 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.398994:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201459.399001:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201459.399034:0:41142:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 16384 00000080:00010000:0.0:1788201459.399040:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e19f2 00000080:00010000:0.0:1788201459.399045:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201459.399048:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e19f2 from it ffffaa1801fcfbf8 00010000:00010000:0.0:1788201459.399052:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e19f2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf391 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.399062:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e19f2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf391 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201459.399093:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201459.399116:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201459.399134:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201459.399144:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e19f2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf391 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201459.399155:0:41142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e19f2 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf391 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201459.399162:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201459.399166:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201459.399169:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e19f2 00000080:00010000:0.0:1788201459.399179:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e19f2 from it ffffaa1801fcfb38 00010000:00010000:0.0:1788201459.399183:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e19f2 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf391 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201459.399190:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e19f2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf391 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201459.399200:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201459.399205:0:41142:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-1-85, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8287b37200, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201459.399214:0:41142:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-85, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:0.0:1788201459.399236:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-85,[0x200006991:0xc8:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201459.399250:0:41142:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e19f2 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbf391 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201459.399330:0:41142:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e19f2 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbf391 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201459.399339:0:41142:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201459.399342:0:41142:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e19f2 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbf391 expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201459.399381:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e19f9 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.399388:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e19f9 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201459.399415:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201459.399424:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8288b92a00 x1875062187443328/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201459.399440:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e19f9 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201459.399462:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b92a00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187443328:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201459.399858:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287e50840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201459.614848:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d717480 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187450624:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201459.614890:0:41142:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e828d715500 x1875062187450368/t77309411825(77309411825) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201475 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201459.614944:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201459.614961:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201459.615031:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1a9a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.615041:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1a9a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.615052:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e1a9a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201459.615073:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe1c00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187450752:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201459.615149:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201459.649204:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d714e00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187451392:192.168.201.121@tcp:101:createmany.0 00010000:00010000:0.0:1788201459.649230:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1aa8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf60e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.649243:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1aa8 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf60e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.649254:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1aa8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf60e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.649266:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1aa8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf60e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.649278:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1aa8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf60e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.649286:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1aa8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf60e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.649294:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1aa8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf60e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201459.649355:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d714e00 x1875062187451392/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201475 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201459.649368:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1aa8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf60e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.649379:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201459.649386:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201459.649424:0:41142:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201459.649430:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1aa8 00000080:00010000:0.0:1788201459.649436:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201459.649438:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1aa8 from it ffffaa1801fcfbf8 00010000:00010000:0.0:1788201459.649441:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1aa8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf60e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.649449:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1aa8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf60e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201459.649483:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201459.649502:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201459.649518:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201459.649527:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1aa8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf60e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201459.649538:0:41142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1aa8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf60e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201459.649555:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201459.649559:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201459.649571:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1aa8 00000080:00010000:0.0:1788201459.649576:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1aa8 from it ffffaa1801fcfb38 00010000:00010000:0.0:1788201459.649579:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1aa8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf60e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201459.649587:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1aa8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf60e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201459.649596:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201459.649601:0:41142:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-1-98, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8287b37980, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201459.649610:0:41142:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-98, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:0.0:1788201459.649629:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-98,[0x200006991:0xd5:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201459.649640:0:41142:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1aa8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbf60e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201459.649698:0:41142:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1aa8 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbf60e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201459.649706:0:41142:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201459.649709:0:41142:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1aa8 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbf60e expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201459.649744:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1aaf lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.649761:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1aaf lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201459.649773:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201459.649780:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8288b91c00 x1875062187451648/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201459.649912:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1aaf lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201459.649935:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b91c00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187451648:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201459.650114:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287e50840 going to sleep for 11 seconds 00000100:00100000:0.0:1788201459.671079:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187452032:192.168.201.121@tcp:101:createmany.0 00010000:00010000:0.0:1788201459.671106:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e1ab6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf63f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.671118:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e1ab6 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf63f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.671141:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e1ab6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf63f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.671153:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e1ab6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf63f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.671165:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e1ab6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf63f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.671184:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e1ab6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf63f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.671191:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e1ab6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf63f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201459.671203:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d716680 x1875062187452032/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201475 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201459.671217:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e1ab6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf63f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.671228:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201459.671234:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201459.671258:0:41142:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201459.671263:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1ab6 00000080:00010000:0.0:1788201459.671268:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201459.671271:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ab6 from it ffffaa1801fcfbf8 00010000:00010000:0.0:1788201459.671275:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e1ab6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf63f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.671283:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e1ab6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf63f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201459.671321:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201459.671339:0:41142:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201459.671353:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201459.671370:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e1ab6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf63f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201459.671381:0:41142:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e1ab6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf63f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201459.671398:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201459.671403:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201459.671406:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1ab6 00000080:00010000:0.0:1788201459.671410:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ab6 from it ffffaa1801fcfb38 00010000:00010000:0.0:1788201459.671413:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e1ab6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf63f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201459.671420:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e1ab6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf63f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201459.671429:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201459.671434:0:41142:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f11b.replay-vbr-1-99, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8287b36a80, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201459.671441:0:41142:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-99, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:0.0:1788201459.671458:0:41142:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-99,[0x200006991:0xd6:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201459.671478:0:41142:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e1ab6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cbf63f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201459.671531:0:41142:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e1ab6 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbf63f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201459.671538:0:41142:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201459.671541:0:41142:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e1ab6 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cbf63f expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201459.671592:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1abd lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.671599:0:41142:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1abd lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201459.671609:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201459.671615:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8288b92680 x1875062187452288/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201459.671624:0:41142:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1abd lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41142 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201459.671643:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b92680 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187452288:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201459.671886:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287e50180 going to sleep for 11 seconds 00000100:00100000:0.0:1788201459.676193:0:41142:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b92680 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187452288:192.168.201.121@tcp:101:createmany.0 00010000:00010000:0.0:1788201459.676222:0:41142:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1abd lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf65b expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.676248:0:41142:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1abd lrc: 4/0,1 mode: --/PR res: [0x200006991:0xd6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf65b expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.676257:0:41142:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1abd lrc: 4/0,1 mode: --/PR res: [0x200006991:0xd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf65b expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.676267:0:41142:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1abd lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf65b expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.676297:0:41142:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1abd lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf65b expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.676328:0:41142:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1abd lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf65b expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.676337:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1abd lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf65b expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.676345:0:41142:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1abd lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cbf65b expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201459.676357:0:41142:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8288b92680 x1875062187452288/t77309411831(77309411831) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201475 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201459.676369:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8288b92680 x1875062187452288/t77309411831(77309411831) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201475 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201459.676383:0:41142:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1abd lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf65b expref: -99 pid: 41142 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201459.676579:0:41142:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1abd lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cbf65b expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.676590:0:41142:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/214 (0 0) 00000080:00200000:0.0:1788201459.676596:0:41142:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201459.676608:0:41142:0:(namei.c:1628:ll_create_it()) VFS Op:name=f11b.replay-vbr-1-99, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:0.0:1788201459.676642:0:41142:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xd6:0x0](ffff8e8289c080c8) 00000080:00200000:0.0:1788201459.676709:0:41142:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xd6:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201459.676714:0:41142:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xd6:0x0](ffff8e8289c080c8) 00000080:00010000:0.0:1788201459.676717:0:41142:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xd6:0x0](ffff8e8289c080c8) for lock 0x9ae3109cee9e1abd 00000080:00010000:0.0:1788201459.676726:0:41142:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xd6:0x0](ffff8e8289c080c8) 00000080:00200008:0.0:1788201459.676737:0:41142:0:(file.c:1090:ll_file_open()) START file f11b.replay-vbr-1-99:[0x200006991:0xd6:0x0](ffff8e8289c080c8), flags 100102 00000002:00100000:0.0:1788201459.676757:0:41142:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8288b92680 x1875062187452288/t77309411831(77309411831) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201475 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201459.676857:0:41142:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1abd from it ffff8e82855fc6c0 00010000:00010000:0.0:1788201459.676863:0:41142:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1abd lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cbf65b expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201459.676873:0:41142:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1abd lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cbf65b expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201459.676893:0:41142:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xd6:0x0](ffff8e8289c080c8), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201459.676921:0:41142:0:(file.c:479:ll_file_release()) START file f11b.replay-vbr-1-99:[0x200006991:0xd6:0x0](ffff8e8289c080c8), flags 100002 00000002:00100000:0.0:1788201459.676967:0:41142:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8288b92680 x1875062187452288/t77309411831(77309411831) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201475 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201459.676977:0:41142:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201459.676981:0:41142:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8288b92300 x1875062187452544/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201459.677000:0:41142:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288b92300 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41142:1875062187452544:192.168.201.121@tcp:35:createmany.0 00000100:00100000:0.0:1788201459.677088:0:41142:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287e50840 going to sleep for 11 seconds 00000080:00200008:1.0:1788201459.972148:0:41162:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8e8289c29988), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201459.972159:0:41162:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c29988),name=/ 00000002:00010000:1.0:1788201459.972182:0:41162:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201459.972245:0:41162:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992200/0x9ae3109cee9e1ac4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41162 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.972256:0:41162:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992200/0x9ae3109cee9e1ac4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41162 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201459.972266:0:41162:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8287992200/0x9ae3109cee9e1ac4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41162 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201459.972291:0:41162:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe2300 pname:cluuid:pid:xid:nid:opc:job umount:6fe158f2-55b2-4a06-bd78-c92321304d95:41162:1875062187452672:192.168.201.121@tcp:101:umount.0 00000100:00100000:1.0:1788201459.972397:0:41162:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1600 going to sleep for 11 seconds 00010000:00010000:1.0:1788201460.018393:0:6173:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788201460.018400:0:6173:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.121@tcp lock: ffff8e828d639400/0x9ae3109cee9e0f72 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0x7cf2114380cbcf2c expref: -99 pid: 41001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201460.018407:0:6173:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8e828d639400 already unused, calling callback (ffffffffc14a1c20) 10000000:00010000:1.0:1788201460.018410:0:6173:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.121@tcp lock: ffff8e828d639400/0x9ae3109cee9e0f72 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0x7cf2114380cbcf2c expref: -99 pid: 41001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201460.018417:0:6173:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.121@tcp lock: ffff8e828d639400/0x9ae3109cee9e0f72 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0x7cf2114380cbcf2c expref: -99 pid: 41001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788201460.018424:0:6173:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.121@tcp lock: ffff8e828d639400/0x9ae3109cee9e0f72 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0x7cf2114380cbcf2c expref: -99 pid: 41001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201460.018429:0:6173:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 10000000:01000000:1.0:1788201460.018431:0:6173:0:(mgc_request.c:944:mgc_blocking_ast()) log params: stopping, won't requeue 10000000:01000000:1.0:1788201460.018433:0:6173:0:(mgc_request.c:130:config_log_put()) dropping config log params 00010000:00010000:1.0:1788201460.018450:0:6173:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.121@tcp lock: ffff8e828d639400/0x9ae3109cee9e0f72 lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x7cf2114380cbcf2c expref: -99 pid: 41001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201460.018456:0:6173:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201460.018473:0:6173:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.121@tcp lock: ffff8e828d639400/0x9ae3109cee9e0f72 lrc: 1/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x7cf2114380cbcf2c expref: -99 pid: 41001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201460.018481:0:6173:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.121@tcp lock: ffff8e828d639400/0x9ae3109cee9e0f72 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x7cf2114380cbcf2c expref: -99 pid: 41001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201460.018491:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201460.018493:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201460.018495:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201460.018514:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187452928:192.168.201.121@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788201460.019448:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201460.024652:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187452928:192.168.201.121@tcp:103:ldlm_bl.0 00000020:00000080:0.0:1788201460.027951:0:20954:0:(genops.c:844:class_export_destroy()) destroying export ffff8e8284ef5800/LOV_OSC_UUID for lustre-OST0001-osc-ffff8e8284ef2800 00000020:00000080:0.0:1788201460.028511:0:20954:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff8e82911ca800 for lustre-OST0001-osc-ffff8e8284ef2800 00000020:00000080:0.0:1788201460.028519:0:20954:0:(genops.c:885:class_export_put()) final put ffff8e82911ce000/6fe158f2-55b2-4a06-bd78-c92321304d95 00000020:00000080:0.0:1788201460.028521:0:20954:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82911ce000/6fe158f2-55b2-4a06-bd78-c92321304d95 for lustre-OST0001-osc-ffff8e8284ef2800 00000020:01000000:0.0:1788201460.028527:0:20954:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0001-osc-ffff8e8284ef2800 (6fe158f2-55b2-4a06-bd78-c92321304d95) 00000020:00000080:0.0:1788201460.034480:0:20954:0:(genops.c:844:class_export_destroy()) destroying export ffff8e8284ef4000/LOV_OSC_UUID for lustre-OST0000-osc-ffff8e8284ef2800 00000020:00000080:1.0:1788201460.037251:0:20954:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff8e82911c8000 for lustre-OST0000-osc-ffff8e8284ef2800 00000020:00000080:1.0:1788201460.037260:0:20954:0:(genops.c:885:class_export_put()) final put ffff8e82911c9800/6fe158f2-55b2-4a06-bd78-c92321304d95 00000020:00000080:1.0:1788201460.037263:0:20954:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82911c9800/6fe158f2-55b2-4a06-bd78-c92321304d95 for lustre-OST0000-osc-ffff8e8284ef2800 00000020:01000000:1.0:1788201460.037275:0:20954:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0000-osc-ffff8e8284ef2800 (6fe158f2-55b2-4a06-bd78-c92321304d95) 00000020:00000080:1.0:1788201460.037528:0:20954:0:(genops.c:844:class_export_destroy()) destroying export ffff8e8284ef2000/6fe158f2-55b2-4a06-bd78-c92321304d95 for lustre-clilov-ffff8e8284ef2800 00000100:00100000:3.0:1788201460.044753:0:41162:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job umount:6fe158f2-55b2-4a06-bd78-c92321304d95:41162:1875062187464704:192.168.201.121@tcp:103:umount.0 00010000:00010000:3.0:1788201460.044759:0:41162:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788201460.044762:0:41162:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788201460.044765:0:41162:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6000/0x9ae3109cee9e13a8 lrc: 0/0,0 mode: --/PR res: [0x2000088d1:0x4a:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbdd6b expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.044791:0:41162:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290f99000/0x9ae3109cee9e1036 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cbd15c expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.044806:0:41162:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290f99000/0x9ae3109cee9e1036 lrc: 1/0,0 mode: --/PR res: [0x2000088d1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbd15c expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.044811:0:41162:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788201460.044815:0:41162:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c900 going to sleep for 0 seconds 00000100:00100000:3.0:1788201460.044819:0:41162:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job umount:6fe158f2-55b2-4a06-bd78-c92321304d95:41162:1875062187464832:192.168.201.121@tcp:103:umount.0 00010000:00010000:3.0:1788201460.044823:0:41162:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788201460.044826:0:41162:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788201460.044828:0:41162:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290f99000/0x9ae3109cee9e1036 lrc: 0/0,0 mode: --/PR res: [0x2000088d1:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbd15c expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.044842:0:41162:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1e00/0x9ae3109cee9e11f6 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cbd77c expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.044857:0:41162:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1e00/0x9ae3109cee9e11f6 lrc: 1/0,0 mode: --/PR res: [0x2000088d1:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbd77c expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.044863:0:41162:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788201460.044867:0:41162:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c900 going to sleep for 0 seconds 00000100:00100000:3.0:1788201460.044871:0:41162:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job umount:6fe158f2-55b2-4a06-bd78-c92321304d95:41162:1875062187464960:192.168.201.121@tcp:103:umount.0 00010000:00010000:3.0:1788201460.044878:0:41162:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788201460.044881:0:41162:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788201460.044884:0:41162:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1e00/0x9ae3109cee9e11f6 lrc: 0/0,0 mode: --/PR res: [0x2000088d1:0x2b:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbd77c expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.044897:0:41162:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13b6 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cbdd9c expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.044913:0:41162:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13b6 lrc: 1/0,0 mode: --/PR res: [0x2000088d1:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbdd9c expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.044918:0:41162:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788201460.044923:0:41162:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c900 going to sleep for 0 seconds 00000100:00100000:3.0:1788201460.044927:0:41162:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job umount:6fe158f2-55b2-4a06-bd78-c92321304d95:41162:1875062187465088:192.168.201.121@tcp:103:umount.0 00010000:00010000:3.0:1788201460.044931:0:41162:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788201460.044934:0:41162:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788201460.044937:0:41162:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6800/0x9ae3109cee9e13b6 lrc: 0/0,0 mode: --/PR res: [0x2000088d1:0x4b:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbdd9c expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.044965:0:41162:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290f99a00/0x9ae3109cee9e1044 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cbd18d expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.044980:0:41162:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290f99a00/0x9ae3109cee9e1044 lrc: 1/0,0 mode: --/PR res: [0x2000088d1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbd18d expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.044987:0:41162:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788201460.044991:0:41162:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c900 going to sleep for 0 seconds 00000100:00100000:3.0:1788201460.044995:0:41162:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job umount:6fe158f2-55b2-4a06-bd78-c92321304d95:41162:1875062187465216:192.168.201.121@tcp:103:umount.0 00010000:00010000:3.0:1788201460.045002:0:41162:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788201460.045005:0:41162:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788201460.045009:0:41162:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290f99a00/0x9ae3109cee9e1044 lrc: 0/0,0 mode: --/PR res: [0x2000088d1:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbd18d expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045021:0:41162:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6200/0x9ae3109cee9e1204 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cbd7ad expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045034:0:41162:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6200/0x9ae3109cee9e1204 lrc: 1/0,0 mode: --/PR res: [0x2000088d1:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbd7ad expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045041:0:41162:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788201460.045045:0:41162:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c900 going to sleep for 0 seconds 00000100:00100000:3.0:1788201460.045049:0:41162:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job umount:6fe158f2-55b2-4a06-bd78-c92321304d95:41162:1875062187465344:192.168.201.121@tcp:103:umount.0 00010000:00010000:3.0:1788201460.045053:0:41162:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788201460.045056:0:41162:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788201460.045059:0:41162:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb6200/0x9ae3109cee9e1204 lrc: 0/0,0 mode: --/PR res: [0x2000088d1:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbd7ad expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045072:0:41162:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7600/0x9ae3109cee9e13c4 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cbddcd expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045086:0:41162:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7600/0x9ae3109cee9e13c4 lrc: 1/0,0 mode: --/PR res: [0x2000088d1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbddcd expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045092:0:41162:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788201460.045096:0:41162:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c900 going to sleep for 0 seconds 00000100:00100000:3.0:1788201460.045100:0:41162:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job umount:6fe158f2-55b2-4a06-bd78-c92321304d95:41162:1875062187465472:192.168.201.121@tcp:103:umount.0 00010000:00010000:3.0:1788201460.045106:0:41162:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788201460.045109:0:41162:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788201460.045112:0:41162:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290eb7600/0x9ae3109cee9e13c4 lrc: 0/0,0 mode: --/PR res: [0x2000088d1:0x4c:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbddcd expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045153:0:41162:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828d639a00/0x9ae3109cee9e1052 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cbd1be expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045169:0:41162:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828d639a00/0x9ae3109cee9e1052 lrc: 1/0,0 mode: --/PR res: [0x2000088d1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbd1be expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045176:0:41162:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788201460.045180:0:41162:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c900 going to sleep for 0 seconds 00000100:00100000:3.0:1788201460.045184:0:41162:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job umount:6fe158f2-55b2-4a06-bd78-c92321304d95:41162:1875062187465600:192.168.201.121@tcp:103:umount.0 00010000:00010000:3.0:1788201460.045189:0:41162:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788201460.045192:0:41162:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788201460.045195:0:41162:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828d639a00/0x9ae3109cee9e1052 lrc: 0/0,0 mode: --/PR res: [0x2000088d1:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbd1be expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045210:0:41162:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290f99800/0x9ae3109cee9e1212 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cbd7de expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045224:0:41162:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290f99800/0x9ae3109cee9e1212 lrc: 1/0,0 mode: --/PR res: [0x2000088d1:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbd7de expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045230:0:41162:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788201460.045234:0:41162:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c900 going to sleep for 0 seconds 00000100:00100000:3.0:1788201460.045238:0:41162:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job umount:6fe158f2-55b2-4a06-bd78-c92321304d95:41162:1875062187465728:192.168.201.121@tcp:103:umount.0 00010000:00010000:3.0:1788201460.045245:0:41162:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788201460.045248:0:41162:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788201460.045251:0:41162:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290f99800/0x9ae3109cee9e1212 lrc: 0/0,0 mode: --/PR res: [0x2000088d1:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbd7de expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045265:0:41162:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1c00/0x9ae3109cee9e13d2 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cbddfe expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045279:0:41162:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1c00/0x9ae3109cee9e13d2 lrc: 1/0,0 mode: --/PR res: [0x2000088d1:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbddfe expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045286:0:41162:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788201460.045289:0:41162:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c900 going to sleep for 0 seconds 00000100:00100000:3.0:1788201460.045293:0:41162:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job umount:6fe158f2-55b2-4a06-bd78-c92321304d95:41162:1875062187465856:192.168.201.121@tcp:103:umount.0 00010000:00010000:3.0:1788201460.045297:0:41162:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788201460.045301:0:41162:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788201460.045303:0:41162:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290ee1c00/0x9ae3109cee9e13d2 lrc: 0/0,0 mode: --/PR res: [0x2000088d1:0x4d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbddfe expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045359:0:41162:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828d639600/0x9ae3109cee9e1060 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cbd1ef expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045376:0:41162:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828d639600/0x9ae3109cee9e1060 lrc: 1/0,0 mode: --/PR res: [0x2000088d1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbd1ef expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045382:0:41162:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788201460.045387:0:41162:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c900 going to sleep for 0 seconds 00000100:00100000:3.0:1788201460.045391:0:41162:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job umount:6fe158f2-55b2-4a06-bd78-c92321304d95:41162:1875062187465984:192.168.201.121@tcp:103:umount.0 00010000:00010000:3.0:1788201460.045398:0:41162:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788201460.045401:0:41162:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788201460.045404:0:41162:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828d639600/0x9ae3109cee9e1060 lrc: 0/0,0 mode: --/PR res: [0x2000088d1:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbd1ef expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045418:0:41162:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290f64a00/0x9ae3109cee9e1220 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cbd80f expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045432:0:41162:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290f64a00/0x9ae3109cee9e1220 lrc: 1/0,0 mode: --/PR res: [0x2000088d1:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbd80f expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045438:0:41162:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788201460.045442:0:41162:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c900 going to sleep for 0 seconds 00000100:00100000:3.0:1788201460.045446:0:41162:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job umount:6fe158f2-55b2-4a06-bd78-c92321304d95:41162:1875062187466112:192.168.201.121@tcp:103:umount.0 00010000:00010000:3.0:1788201460.045451:0:41162:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788201460.045454:0:41162:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788201460.045457:0:41162:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e8290f64a00/0x9ae3109cee9e1220 lrc: 0/0,0 mode: --/PR res: [0x2000088d1:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbd80f expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045471:0:41162:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828d790e00/0x9ae3109cee9e13e0 lrc: 2/0,0 mode: PR/PR res: [0x2000088d1:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cbde2f expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045485:0:41162:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828d790e00/0x9ae3109cee9e13e0 lrc: 1/0,0 mode: --/PR res: [0x2000088d1:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbde2f expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201460.045491:0:41162:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788201460.045494:0:41162:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c900 going to sleep for 0 seconds 00000100:00100000:3.0:1788201460.045498:0:41162:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job umount:6fe158f2-55b2-4a06-bd78-c92321304d95:41162:1875062187466240:192.168.201.121@tcp:103:umount.0 00010000:00010000:3.0:1788201460.045504:0:41162:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788201460.045507:0:41162:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788201460.045510:0:41162:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8284ef2800 lock: ffff8e828d790e00/0x9ae3109cee9e13e0 lrc: 0/0,0 mode: --/PR res: [0x2000088d1:0x4e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cbde2f expref: -99 pid: 41043 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000100:00080000:3.0:1788201460.046364:0:41162:0:(import.c:65:import_set_state_nolock()) ffff8e82911cb800 lustre-MDT0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:3.0:1788201460.049685:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82911cb800 lustre-MDT0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:3.0:1788201460.049718:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6fe158f2-55b2-4a06-bd78-c92321304d95:2400:1875062187466368:192.168.201.121@tcp:39:umount.0 00000100:00080000:1.0:1788201460.049772:0:41162:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:1.0:1788201460.049776:0:41162:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8284ef2800: committing for last_committed 77309411628 gen 2 00000100:00100000:1.0:1788201460.049786:0:41162:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8e82894db800 x1875062187385472/t77309411631(77309411631) o36->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201466 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201460.049806:0:41162:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8e828d720000 x1875062187385984/t77309411632(77309411632) o36->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201466 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201460.049820:0:41162:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8e828d716680 x1875062187466368/t77309411833(77309411833) o39->lustre-MDT0000-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201470 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201460.049833:0:41162:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788201460.053591:0:41162:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x9ae3109cee9e0f80 00000020:00000080:1.0:1788201460.053607:0:41162:0:(genops.c:885:class_export_put()) final put ffff8e82856e7000/6fe158f2-55b2-4a06-bd78-c92321304d95 00000100:00080000:1.0:1788201460.053757:0:41162:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 6fe158f2-55b2-4a06-bd78-c92321304d95->lustre-MDT0001_UUID 00000100:00080000:1.0:1788201460.057844:0:41162:0:(import.c:65:import_set_state_nolock()) ffff8e82911cc000 lustre-MDT0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788201460.057906:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6fe158f2-55b2-4a06-bd78-c92321304d95:2400:1875062187466496:192.168.201.121@tcp:39:umount.0 00000020:00000080:0.0:1788201460.059119:0:39443:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82856e7000/6fe158f2-55b2-4a06-bd78-c92321304d95 for lustre-MDT0000-mdc-ffff8e8284ef2800 00000100:00080000:3.0:1788201460.063119:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82911cc000 lustre-MDT0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:3.0:1788201460.063187:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6fe158f2-55b2-4a06-bd78-c92321304d95:2400:1875062187466496:192.168.201.121@tcp:39:umount.0 00000100:00080000:1.0:1788201460.063857:0:41162:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:1.0:1788201460.063865:0:41162:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8e8284ef2800: committing for last_committed 0 gen 2 00000100:00100000:1.0:1788201460.063878:0:41162:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8e8290fe2300 x1875062187466496/t4294967339(4294967339) o39->lustre-MDT0001-mdc-ffff8e8284ef2800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201470 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201460.063902:0:41162:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788201460.065640:0:41162:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x9ae3109cee9e0f87 00000020:00000080:1.0:1788201460.065656:0:41162:0:(genops.c:885:class_export_put()) final put ffff8e82879e3000/6fe158f2-55b2-4a06-bd78-c92321304d95 00000020:00000080:1.0:1788201460.065695:0:41162:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x9ae3109cee9e0f79 00000020:00000080:1.0:1788201460.065700:0:41162:0:(genops.c:885:class_export_put()) final put ffff8e82856e3000/6fe158f2-55b2-4a06-bd78-c92321304d95 00000020:01000000:1.0:1788201460.065710:0:41162:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff8e8284ef2800 (flags='') 00000020:00000080:1.0:1788201460.065719:0:41162:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788201460.065726:0:41162:0:(obd_config.c:956:class_cleanup()) lustre-clilov-ffff8e8284ef2800: forcing exports to disconnect: 0/1 00000020:00080000:1.0:1788201460.065730:0:41162:0:(genops.c:1558:class_disconnect_exports()) OBD device 7 (ffff8e8283ee19b8) has no exports 00000020:00000080:1.0:1788201460.065816:0:41162:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788201460.065824:0:41162:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff8e8284ef2800 (uuid 6fe158f2-55b2-4a06-bd78-c92321304d95) 00000020:00000080:1.0:1788201460.065826:0:41162:0:(genops.c:885:class_export_put()) final put ffff8e82856e2000/6fe158f2-55b2-4a06-bd78-c92321304d95 00000020:00000080:1.0:1788201460.065829:0:41162:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82856e2000/6fe158f2-55b2-4a06-bd78-c92321304d95 for lustre-clilov-ffff8e8284ef2800 00000020:01000000:1.0:1788201460.065834:0:41162:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff8e8284ef2800 (6fe158f2-55b2-4a06-bd78-c92321304d95) 00000020:01000000:1.0:1788201460.065978:0:41162:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff8e8284ef2800 (flags='') 00000020:00000080:1.0:1788201460.065982:0:41162:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788201460.065987:0:41162:0:(obd_config.c:956:class_cleanup()) lustre-clilmv-ffff8e8284ef2800: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788201460.065989:0:41162:0:(genops.c:1558:class_disconnect_exports()) OBD device 8 (ffff8e8283e9cd28) has no exports 00000020:00000080:1.0:1788201460.066031:0:41162:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788201460.066035:0:41162:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff8e8284ef2800 (uuid 6fe158f2-55b2-4a06-bd78-c92321304d95) 00000020:01000000:1.0:1788201460.066038:0:41162:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff8e8284ef2800 (flags='') 00000020:00000080:1.0:1788201460.066041:0:41162:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788201460.066043:0:41162:0:(obd_config.c:956:class_cleanup()) lustre-MDT0000-mdc-ffff8e8284ef2800: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788201460.066046:0:41162:0:(genops.c:1558:class_disconnect_exports()) OBD device 9 (ffff8e8283e9b370) has no exports 00000020:00000080:0.0:1788201460.066636:0:39443:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82879e3000/6fe158f2-55b2-4a06-bd78-c92321304d95 for lustre-MDT0001-mdc-ffff8e8284ef2800 00000020:00000080:0.0:1788201460.066652:0:39443:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82856e3000/6fe158f2-55b2-4a06-bd78-c92321304d95 for lustre-clilmv-ffff8e8284ef2800 00000020:00000080:0.0:1788201460.066656:0:39443:0:(genops.c:885:class_export_put()) final put ffff8e82911cb000/6fe158f2-55b2-4a06-bd78-c92321304d95 00000020:00000080:0.0:1788201460.066658:0:39443:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82911cb000/6fe158f2-55b2-4a06-bd78-c92321304d95 for lustre-clilmv-ffff8e8284ef2800 00000020:01000000:0.0:1788201460.066662:0:39443:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilmv-ffff8e8284ef2800 (6fe158f2-55b2-4a06-bd78-c92321304d95) 00000008:01000000:3.0:1788201460.066867:0:41162:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff8e8284ef2800: client import never connected 00000100:00080000:3.0:1788201460.066872:0:41162:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:3.0:1788201460.066877:0:41162:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8284ef2800: committing for last_committed 77309411628 gen 3 00000100:00100000:3.0:1788201460.066884:0:41162:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:3.0:1788201460.069350:0:41162:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:3.0:1788201460.069367:0:41162:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0000-mdc-ffff8e8284ef2800 (uuid 6fe158f2-55b2-4a06-bd78-c92321304d95) 00000020:01000000:3.0:1788201460.069373:0:41162:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0001-mdc-ffff8e8284ef2800 (flags='') 00000020:00000080:0.0:1788201460.069608:0:39443:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff8e82911cb800 for lustre-MDT0000-mdc-ffff8e8284ef2800 00000020:00000080:0.0:1788201460.069618:0:39443:0:(genops.c:885:class_export_put()) final put ffff8e82911cf000/6fe158f2-55b2-4a06-bd78-c92321304d95 00000020:00000080:0.0:1788201460.069622:0:39443:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82911cf000/6fe158f2-55b2-4a06-bd78-c92321304d95 for lustre-MDT0000-mdc-ffff8e8284ef2800 00000020:01000000:0.0:1788201460.069628:0:39443:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0000-mdc-ffff8e8284ef2800 (6fe158f2-55b2-4a06-bd78-c92321304d95) 00000020:00000080:3.0:1788201460.070546:0:41162:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:3.0:1788201460.070564:0:41162:0:(obd_config.c:956:class_cleanup()) lustre-MDT0001-mdc-ffff8e8284ef2800: forcing exports to disconnect: 0/2 00000020:00080000:3.0:1788201460.070569:0:41162:0:(genops.c:1558:class_disconnect_exports()) OBD device 10 (ffff8e8283e999b8) has no exports 00000008:01000000:3.0:1788201460.070793:0:41162:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0001-mdc-ffff8e8284ef2800: client import never connected 00000100:00080000:3.0:1788201460.070797:0:41162:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:3.0:1788201460.070801:0:41162:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8e8284ef2800: committing for last_committed 0 gen 3 00000100:00100000:3.0:1788201460.070808:0:41162:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:3.0:1788201460.073157:0:41162:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:3.0:1788201460.073170:0:41162:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0001-mdc-ffff8e8284ef2800 (uuid 6fe158f2-55b2-4a06-bd78-c92321304d95) 00000080:02000400:3.0:1788201460.073175:0:41162:0:(llite_lib.c:1739:ll_put_super()) Unmounted lustre-client 00000020:01000000:3.0:1788201460.073177:0:41162:0:(obd_config.c:1198:class_del_profile()) Del profile lustre-client 00000020:01000004:3.0:1788201460.073235:0:41162:0:(obd_mount.c:897:lustre_common_put_super()) dropping sb ffff8e8284ef2800 00000020:01000004:3.0:1788201460.073238:0:41162:0:(obd_mount.c:578:lustre_stop_mgc()) MGC still has 1 references. 00000020:01000004:3.0:1788201460.073241:0:41162:0:(obd_mount.c:910:lustre_common_put_super()) MGC still in use 00000020:01000004:3.0:1788201460.073242:0:41162:0:(obd_mount.c:710:lustre_put_lsi()) put ffff8e8284ef2800 1 00000020:01000004:3.0:1788201460.073245:0:41162:0:(obd_mount.c:668:lustre_free_lsi()) Freeing lsi ffff8e8284ef3000 00000020:00000080:1.0:1788201460.073705:0:20954:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff8e82911cc000 for lustre-MDT0001-mdc-ffff8e8284ef2800 00000020:00000080:1.0:1788201460.073715:0:20954:0:(genops.c:885:class_export_put()) final put ffff8e82911cd800/6fe158f2-55b2-4a06-bd78-c92321304d95 00000020:00000080:1.0:1788201460.073718:0:20954:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82911cd800/6fe158f2-55b2-4a06-bd78-c92321304d95 for lustre-MDT0001-mdc-ffff8e8284ef2800 00000020:01000000:1.0:1788201460.073724:0:20954:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0001-mdc-ffff8e8284ef2800 (6fe158f2-55b2-4a06-bd78-c92321304d95) 00010000:00010000:1.0:1788201460.932521:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201461.958402:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201462.277273:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187466624:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201462.277414:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187466880:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201462.277442:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187467136:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201462.277561:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788201462.277568:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187466752:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201462.277593:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187467008:192.168.201.121@tcp:400:kworker.0 00000100:00080000:1.0:1788201462.281751:0:2402:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff8e8282a4b800 of lustre-MDT0000_UUID@192.168.201.121@tcp abruptly disconnected: reconnecting 00000100:00080000:1.0:1788201462.281760:0:2402:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:1.0:1788201462.281767:0:2402:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8e8282a4b800: Connection to lustre-MDT0000 (at 192.168.201.121@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:1.0:1788201462.281773:0:2402:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201462.281778:0:2402:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 2821 00000100:00080000:1.0:1788201462.281787:0:2402:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201462.281796:0:2402:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201462.281825:0:2402:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289513800 x1875062187467264/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201462.282145:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187466752:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201462.282168:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894db800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187467008:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201462.282180:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187466880:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201462.282192:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187467136:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201462.282230:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187467264:192.168.201.121@tcp:38:ptlrpcd_00_01.0 00000100:00080000:3.0:1788201462.298681:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201462.298696:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-19) 00000100:00100000:3.0:1788201462.298702:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289513800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187467264:192.168.201.121@tcp:38:ptlrpcd_00_01.0 00010000:00010000:0.0:1788201462.985476:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201464.011598:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201465.028635:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201466.052656:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201467.076674:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201467.396468:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201467.396475:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201467.396479:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201467.396490:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 2922 00000100:00080000:1.0:1788201467.396501:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1788201467.396509:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201467.396518:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201467.396537:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289512300 x1875062187467520/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201467.396910:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187467520:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788201467.397103:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187467648:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201467.397140:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187467904:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201467.397169:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788201467.397175:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187467392:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201467.397202:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187467776:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201467.403043:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187467776:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201467.403061:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289513480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187467648:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201467.403072:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289513100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187467904:192.168.201.121@tcp:400:kworker.0 00000100:00080000:3.0:1788201467.403159:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201467.403165:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-19) 00000100:00100000:3.0:1788201467.403169:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187467520:192.168.201.121@tcp:38:kworker.0 00010000:00010000:3.0:1788201468.100829:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201469.124745:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201470.148687:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201471.174516:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201472.197073:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201472.516823:0:39443:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201472.516842:0:39443:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201472.516846:0:39443:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201472.516856:0:39443:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 2927 00000100:00080000:0.0:1788201472.516866:0:39443:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1788201472.516874:0:39443:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201472.516885:0:39443:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201472.516905:0:39443:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82910c8e00 x1875062187468160/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201472.516976:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187468288:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201472.517068:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187468544:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201472.517101:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1788201472.517108:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910ca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187468032:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201472.517134:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187468416:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201472.517236:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187468160:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201472.523032:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187468416:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201472.523065:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187468288:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201472.523078:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187468544:192.168.201.121@tcp:400:kworker.0 00000100:00080000:3.0:1788201472.523484:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201472.523492:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-19) 00000100:00100000:3.0:1788201472.523496:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187468160:192.168.201.121@tcp:38:kworker.0 00010000:00010000:1.0:1788201473.221579:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201474.245077:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201475.268663:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201476.293045:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201477.316901:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201477.636770:0:39443:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201477.636777:0:39443:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201477.636781:0:39443:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201477.636803:0:39443:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 2932 00000100:00080000:0.0:1788201477.636815:0:39443:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 15s 00000100:00100000:1.0:1788201477.636819:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00080000:0.0:1788201477.636824:0:39443:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788201477.636835:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187468672:192.168.201.121@tcp:400:kworker.0 00000008:00100000:0.0:1788201477.636836:0:39443:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201477.636860:0:39443:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82910cb800 x1875062187468800/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201477.636944:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187468928:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201477.636972:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187469184:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201477.637007:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788201477.637012:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187469056:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201477.637153:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187468800:192.168.201.121@tcp:38:kworker.0 00000100:00080000:3.0:1788201477.646020:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201477.646035:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-19) 00000100:00100000:3.0:1788201477.646040:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187468800:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201477.646130:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187469056:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201477.646146:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187468928:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201477.646157:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187469184:192.168.201.121@tcp:400:kworker.0 00010000:00010000:3.0:1788201478.345460:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1788201478.660485:0:2402:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201462/real 1788201462] req@ffff8e8290fe2300 x1875062187466624/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201478 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201478.729565:0:2402:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8e8290fe2300 x1875062187466624/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201478 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201478.729638:0:2402:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from FULL to DISCONN 00000100:02020000:0.0:1788201478.729644:0:2402:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail 00000100:00080000:0.0:1788201478.729649:0:2402:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.201.121@tcp for MGC192.168.201.121@tcp not replayable, auto-deactivating 00000100:00080000:0.0:1788201478.729651:0:2402:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:0.0:1788201478.729657:0:2402:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8e8290fe2300 x1875062187466624/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201478 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201478.729668:0:2402:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8e8289512d80 x1875062187467392/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201483 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201478.729677:0:2402:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8e82910ca300 x1875062187468032/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201488 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201478.729712:0:2402:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8e82910c9500 x1875062187468672/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201493 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:0.0:1788201478.729723:0:2402:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:0.0:1788201478.729726:0:2402:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:0.0:1788201478.729727:0:2402:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:0.0:1788201478.729730:0:2402:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:0.0:1788201478.729774:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187468672:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201478.729785:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910ca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187468032:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201478.729793:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187467392:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201478.729799:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187466624:192.168.201.121@tcp:400:kworker.0 00000100:00080000:1.0:1788201478.730099:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201478.730106:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201478.730110:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201478.730117:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 2818 00000100:00080000:1.0:1788201478.730134:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788201478.730176:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289510a80 x1875062187469312/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201478.730210:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201478.730213:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201478.730215:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201478.730220:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 2937 00000100:00080000:1.0:1788201478.730225:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201478.730239:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201478.730250:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201478.730264:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289513b80 x1875062187469440/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201478.730359:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187469568:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201478.730466:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187469824:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201478.730494:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201478.730510:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187469696:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201478.731851:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187469312:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788201478.731946:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187469440:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788201478.733185:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187469696:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201478.733206:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187469824:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201478.737065:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187469568:192.168.201.121@tcp:400:kworker.0 00000100:00080000:3.0:1788201478.737744:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201478.737756:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-19) 00000100:00100000:3.0:1788201478.737762:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289513b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187469440:192.168.201.121@tcp:38:kworker.0 00010000:00010000:3.0:1788201479.367700:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201480.399633:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201481.414831:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201482.436510:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201483.462648:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201483.785252:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201483.785263:0:20954:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788201483.785266:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201483.785269:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201483.785273:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201483.785285:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 2938 00000100:00080000:3.0:1788201483.785295:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201483.785302:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201483.785374:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201483.785406:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d714a80 x1875062187469952/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201483.785470:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d714a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187469952:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201483.785642:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d715880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187470208:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201483.785678:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201483.785684:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187470080:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201483.785704:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d714e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187470336:192.168.201.121@tcp:400:kworker.0 00000100:00080000:1.0:1788201483.797345:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201483.797359:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-19) 00000100:00100000:1.0:1788201483.797365:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d714a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187469952:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201483.797555:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187470080:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201483.797568:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d714e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187470336:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201483.797578:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d715880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187470208:192.168.201.121@tcp:400:kworker.0 00010000:00010000:3.0:1788201484.484613:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201485.509032:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201486.536519:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201487.556569:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201488.582379:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201488.900483:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201488.900495:0:20954:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788201488.900498:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201488.900501:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201488.900506:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201488.900518:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 2943 00000100:00080000:3.0:1788201488.900530:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201488.900537:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201488.900549:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201488.900586:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d715880 x1875062187470464/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201488.900785:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201478/real 1788201478] req@ffff8e8289510a80 x1875062187469312/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201488 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201488.900812:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289510a80 x1875062187469312/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201488 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201488.900833:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d715880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187470464:192.168.201.121@tcp:38:kworker.0 00000100:00080000:0.0:1788201488.900936:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201488.901066:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201488.901071:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187469312:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788201488.906237:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201488.906248:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201488.906252:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201488.906262:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 2938 00000100:00080000:3.0:1788201488.906274:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788201488.906281:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788201488.906354:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289529c00 x1875062187470976/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201488.908363:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d714e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187470592:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201488.917747:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187470720:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201488.917763:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d714a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187470848:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201488.920568:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187470976:192.168.201.121@tcp:250:kworker.0 00010000:00010000:3.0:1788201489.604583:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201490.628639:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:0.0:1788201490.756437:0:6175:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:0.0:1788201490.756449:0:6175:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8e8282a4b800 from 75 10000000:01000000:0.0:1788201490.756455:0:6175:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:0.0:1788201490.756509:0:6175:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1400/0x9ae3109cee9e1acb lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201490.756521:0:6175:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1400/0x9ae3109cee9e1acb lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201490.756531:0:6175:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1400/0x9ae3109cee9e1acb lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201490.756558:0:6175:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910ca300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187471104:192.168.201.121@tcp:101:ll_cfg_requeue.0 00000100:00100000:0.0:1788201490.756661:0:6175:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:1.0:1788201490.760891:0:6175:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910ca300 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187471104:192.168.201.121@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201490.760924:0:6175:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1400/0x9ae3109cee9e1acb lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cbf884 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201490.760937:0:6175:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1400/0x9ae3109cee9e1acb lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x7cf2114380cbf884 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201490.761039:0:6175:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187471232:192.168.201.121@tcp:256:ll_cfg_requeue.0 00000100:00100000:1.0:1788201490.761123:0:6175:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201490.768349:0:6175:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187471232:192.168.201.121@tcp:256:ll_cfg_requeue.0 00000100:00080000:1.0:1788201490.768417:0:6175:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8e82871d2800 already not connected (conn 144, was 0): DISCONN 00010000:00080000:1.0:1788201490.768430:0:6175:0:(ldlm_lib.c:85:import_set_conn()) imp ffff8e82871d2800@lustre-MDT0000-mdc-ffff8e8282a4b800: found existing conn 192.168.201.121@tcp, moved to head 00000100:00080000:1.0:1788201490.768436:0:6175:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201490.768444:0:6175:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201490.768470:0:6175:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201490.768498:0:6175:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289510700 x1875062187471360/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 10000000:01000000:1.0:1788201490.768556:0:6175:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.121@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:1.0:1788201490.768566:0:6175:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1400/0x9ae3109cee9e1acb lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cbf884 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201490.768577:0:6175:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1400/0x9ae3109cee9e1acb lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cbf884 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201490.768585:0:6175:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:1.0:1788201490.768588:0:6175:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff8e8282a4b800 from 52 10000000:01000000:1.0:1788201490.768591:0:6175:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:1.0:1788201490.768619:0:6175:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8287992000/0x9ae3109cee9e1ad2 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201490.768626:0:6175:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.121@tcp lock: ffff8e8287992000/0x9ae3109cee9e1ad2 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201490.768635:0:6175:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.121@tcp lock: ffff8e8287992000/0x9ae3109cee9e1ad2 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201490.768654:0:6175:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187471488:192.168.201.121@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201490.768719:0:6175:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1840 going to sleep for 11 seconds 00000100:00100000:0.0:1788201490.769153:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187471360:192.168.201.121@tcp:38:ll_cfg_requeue.0 00000100:00100000:1.0:1788201490.773890:0:6175:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187471488:192.168.201.121@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201490.773912:0:6175:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.121@tcp lock: ffff8e8287992000/0x9ae3109cee9e1ad2 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cbf88b expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201490.773935:0:6175:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.121@tcp lock: ffff8e8287992000/0x9ae3109cee9e1ad2 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x7cf2114380cbf88b expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201490.773997:0:6175:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187471616:192.168.201.121@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201490.774042:0:6175:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201490.783275:0:6175:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187471616:192.168.201.121@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201490.783377:0:6175:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187471744:192.168.201.121@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788201490.783478:0:6175:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201490.786359:0:6175:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187471744:192.168.201.121@tcp:503:ll_cfg_requeue.0 00000020:01000000:0.0:1788201490.797191:0:6175:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 52-51 (rc=0) 10000000:01000000:0.0:1788201490.797227:0:6175:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.121@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:0.0:1788201490.797241:0:6175:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8287992000/0x9ae3109cee9e1ad2 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cbf88b expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201490.797253:0:6175:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.121@tcp lock: ffff8e8287992000/0x9ae3109cee9e1ad2 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cbf88b expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201490.797283:0:6175:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:0.0:1788201490.797285:0:6175:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff8e8282a4b800 from 7 10000000:01000000:0.0:1788201490.797288:0:6175:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:0.0:1788201490.798046:0:6175:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1e00/0x9ae3109cee9e1ad9 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201490.798058:0:6175:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1e00/0x9ae3109cee9e1ad9 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201490.798069:0:6175:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1e00/0x9ae3109cee9e1ad9 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201490.798099:0:6175:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187471872:192.168.201.121@tcp:101:ll_cfg_requeue.0 00000100:00100000:0.0:1788201490.798258:0:6175:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:1.0:1788201490.808998:0:6175:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187471872:192.168.201.121@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201490.809039:0:6175:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1e00/0x9ae3109cee9e1ad9 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cbf892 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201490.809053:0:6175:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1e00/0x9ae3109cee9e1ad9 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x7cf2114380cbf892 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201490.809144:0:6175:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187472000:192.168.201.121@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201490.809230:0:6175:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201490.811629:0:6175:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187472000:192.168.201.121@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201490.811687:0:6175:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187472128:192.168.201.121@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788201490.811776:0:6175:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201490.814197:0:6175:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187472128:192.168.201.121@tcp:503:ll_cfg_requeue.0 00000020:01000000:1.0:1788201490.815505:0:6175:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:1.0:1788201490.815524:0:6175:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.121@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:1.0:1788201490.815536:0:6175:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1e00/0x9ae3109cee9e1ad9 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cbf892 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201490.815548:0:6175:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1e00/0x9ae3109cee9e1ad9 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cbf892 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788201490.815557:0:6175:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:1.0:1788201490.815560:0:6175:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff8e82871a8000 from 1 10000000:01000000:1.0:1788201490.815563:0:6175:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:1.0:1788201490.815598:0:6175:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8287992200/0x9ae3109cee9e1ae0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201490.815606:0:6175:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.121@tcp lock: ffff8e8287992200/0x9ae3109cee9e1ae0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201490.815616:0:6175:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.121@tcp lock: ffff8e8287992200/0x9ae3109cee9e1ae0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201490.815639:0:6175:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187472256:192.168.201.121@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788201490.815729:0:6175:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201490.818367:0:6175:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187472256:192.168.201.121@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201490.818395:0:6175:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.121@tcp lock: ffff8e8287992200/0x9ae3109cee9e1ae0 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cbf899 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201490.818405:0:6175:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.121@tcp lock: ffff8e8287992200/0x9ae3109cee9e1ae0 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x7cf2114380cbf899 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201490.818486:0:6175:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187472384:192.168.201.121@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201490.818591:0:6175:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201490.831494:0:6175:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187472384:192.168.201.121@tcp:501:ll_cfg_requeue.0 10000000:01000000:1.0:1788201490.831544:0:6175:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.121@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:1.0:1788201490.831555:0:6175:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8287992200/0x9ae3109cee9e1ae0 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cbf899 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201490.831567:0:6175:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.121@tcp lock: ffff8e8287992200/0x9ae3109cee9e1ae0 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cbf899 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201491.652676:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201492.676567:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201493.703139:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201494.021491:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 1 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201494.021498:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:0.0:1788201494.021609:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187472512:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201494.021720:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187472768:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201494.021754:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1788201494.021760:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187472640:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201494.021781:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187472896:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201494.036371:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187472640:192.168.201.121@tcp:400:kworker.0 00010000:00010000:1.0:1788201494.036423:0:2402:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 1 blwis in pool 00000100:00100000:1.0:1788201494.036431:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187472896:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201494.036444:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187472512:192.168.201.121@tcp:400:kworker.0 00010000:00010000:1.0:1788201494.036460:0:2402:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 2 blwis in pool 00000100:00100000:1.0:1788201494.036465:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187472768:192.168.201.121@tcp:400:kworker.0 00010000:00010000:3.0:1788201494.036511:0:6173:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (1 blwis in pool) 00010000:00010000:3.0:1788201494.036540:0:6173:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201494.036546:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201494.036547:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201494.036549:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201494.036686:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201494.724610:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201495.044575:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201495.044586:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:1.0:1788201495.044741:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187473152:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201495.044929:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187473024:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201495.045582:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187473280:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201495.047929:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187473408:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201495.073790:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187473024:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201495.073818:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187473280:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201495.074013:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187473152:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201495.074026:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187473408:192.168.201.121@tcp:400:kworker.0 00010000:00010000:3.0:1788201495.748619:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201496.772461:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:0.0:1788201496.983989:0:41485:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8e8292531988), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201496.984002:0:41485:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201496.984024:0:41485:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201496.984086:0:41485:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41485 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201496.984096:0:41485:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41485 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201496.984107:0:41485:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41485 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1788201496.984130:0:41485:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != REPLAY) req@ffff8e82910cb100 x1875062187473536/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788201496.984142:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:0.0:1788201497.801232:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201497.988630:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:3.0:1788201498.824368:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201499.013089:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201499.844802:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201500.039567:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00000100:00080000:0.0:1788201500.165625:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201500.165634:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:1.0:1788201500.165750:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187473664:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201500.165894:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187473920:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201500.165936:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788201500.165942:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187473792:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201500.165968:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187474048:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201500.184786:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187473792:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201500.184811:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187473664:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201500.188789:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187473920:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201500.189352:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187474048:192.168.201.121@tcp:400:kworker.0 00010000:00010000:0.0:1788201500.871707:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201501.060470:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:3.0:1788201501.892652:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201502.084467:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201502.919264:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201503.110274:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:0.0:1788201503.941022:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201504.135134:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:3.0:1788201504.967299:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201505.156472:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00000100:00080000:0.0:1788201505.284997:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201505.285006:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:1.0:1788201505.285140:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187474304:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201505.285297:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187474560:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201505.285396:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788201505.285403:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910ca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187474176:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201505.285430:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187474432:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201505.296282:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910ca300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187474176:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201505.296346:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187474304:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201505.296360:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187474560:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201505.296745:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187474432:192.168.201.121@tcp:400:kworker.0 00010000:00010000:1.0:1788201505.992182:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201506.180807:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:0.0:1788201507.012655:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201507.204509:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201508.036643:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201508.228762:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201509.061155:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201509.254490:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:0.0:1788201510.086306:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201510.276764:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00000100:00080000:1.0:1788201510.409144:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201510.409153:0:20954:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:1.0:1788201510.409207:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187474816:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201510.409377:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187475072:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201510.409420:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788201510.409425:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187474688:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201510.409452:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187474944:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201510.413767:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187474688:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201510.413796:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187474944:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201510.413808:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187474816:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201510.413819:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187475072:192.168.201.121@tcp:400:kworker.0 00010000:00010000:1.0:1788201511.108768:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201511.301180:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201512.132716:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201512.324965:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:0.0:1788201513.160088:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201513.348881:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201514.183155:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201514.374135:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201515.205829:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201515.400157:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00000100:00100000:3.0:1788201515.524905:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2403:1875062187475200:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201515.525096:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187475456:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201515.525142:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788201515.525149:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187475328:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201515.525179:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187475584:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201515.538657:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187475328:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201515.538680:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187475584:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201515.538691:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2403:1875062187475200:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201515.538702:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187475456:192.168.201.121@tcp:400:kworker.0 00010000:00010000:0.0:1788201516.228797:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201516.422928:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201517.252804:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201517.444479:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201518.276559:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201518.468477:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:0.0:1788201519.303616:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201519.492427:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201520.330150:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201520.517394:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00000100:00080000:1.0:1788201520.645337:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201520.645343:0:20954:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:0.0:1788201520.648868:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187475712:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201520.649001:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187475968:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201520.649035:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1788201520.649042:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187475840:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201520.649079:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187476096:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201520.657613:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187476096:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201520.657628:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187475712:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201520.657639:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187475968:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201520.659246:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187475840:192.168.201.121@tcp:400:kworker.0 00010000:00010000:1.0:1788201521.352071:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201521.544444:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:0.0:1788201522.372776:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201522.564723:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201523.396657:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201523.588532:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201524.420663:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201524.612997:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:3.0:1788201525.447050:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201525.638372:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00000100:00080000:0.0:1788201525.765454:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201525.765464:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:0.0:1788201525.765517:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187476352:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201525.765658:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187476608:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201525.765708:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:0.0:1788201525.765715:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187476224:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201525.765743:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187476480:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201525.775680:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187476352:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201525.778408:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187476224:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201525.783284:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187476480:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201525.783304:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187476608:192.168.201.121@tcp:400:kworker.0 00010000:00010000:1.0:1788201526.468574:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201526.660495:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201527.493952:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201527.685125:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:3.0:1788201528.518301:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201528.708504:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201529.551523:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201529.732899:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201530.564754:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201530.757098:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00000100:00080000:0.0:1788201530.884568:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201530.884576:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:0.0:1788201530.884657:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187476864:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201530.884795:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187477120:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201530.884835:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:0.0:1788201530.884841:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187476736:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201530.884870:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187476992:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201530.887445:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187476736:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201530.887469:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187476992:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201530.887480:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187476864:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201530.887491:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187477120:192.168.201.121@tcp:400:kworker.0 00010000:00010000:0.0:1788201531.588513:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201531.780637:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201532.612546:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201532.804471:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201533.636609:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201533.830599:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:0.0:1788201534.660709:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201534.853356:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201535.684678:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201535.876468:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00000100:00080000:1.0:1788201536.012394:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201536.012404:0:20954:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:0.0:1788201536.012532:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187477376:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201536.012685:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187477632:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201536.012726:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:0.0:1788201536.012733:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187477248:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201536.012762:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187477504:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201536.014782:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187477248:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201536.022000:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187477504:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201536.022019:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187477376:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201536.022031:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187477632:192.168.201.121@tcp:400:kworker.0 00010000:00010000:1.0:1788201536.709509:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201536.900896:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:0.0:1788201537.735148:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201537.924484:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201538.756650:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201538.948462:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201539.782882:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201539.973499:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:0.0:1788201540.805908:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201540.996458:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00000100:00080000:0.0:1788201541.125419:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201541.125426:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:0.0:1788201541.125475:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187477888:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201541.125576:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187478144:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201541.125609:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:0.0:1788201541.125615:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187477760:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201541.125641:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187478016:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201541.132754:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c8a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187478016:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201541.132771:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187477888:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201541.132782:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187478144:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201541.135038:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187477760:192.168.201.121@tcp:400:kworker.0 00010000:00010000:1.0:1788201541.831685:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201542.020513:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201542.852875:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201543.047922:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:0.0:1788201543.876755:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201544.069255:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201544.910951:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201545.092621:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201545.924664:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201546.116709:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00000100:00080000:3.0:1788201546.245437:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201546.245447:0:20954:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:3.0:1788201546.245528:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289528700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187478400:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201546.245682:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187478656:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201546.246446:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201546.246457:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2404:1875062187478272:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201546.246509:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187478528:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201546.260874:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2404:1875062187478272:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201546.260906:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289528700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187478400:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201546.260919:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187478656:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201546.266483:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187478528:192.168.201.121@tcp:400:kworker.0 00010000:00010000:0.0:1788201546.948718:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201547.141093:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201547.972531:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201548.165021:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201549.000952:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201549.188439:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:0.0:1788201550.031139:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201550.213769:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201551.044996:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201551.236552:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00000100:00080000:1.0:1788201551.368167:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201551.368191:0:20954:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:0.0:1788201551.368371:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187478784:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201551.368466:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187479040:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201551.368500:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1788201551.368507:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187478912:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201551.368531:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187479168:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201551.371484:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187479168:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201551.371508:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289513b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187479040:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201551.374155:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187478912:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201551.377847:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187478784:192.168.201.121@tcp:400:kworker.0 00010000:00010000:1.0:1788201552.069230:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201552.260447:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:0.0:1788201553.092848:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201553.284497:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201554.117810:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201554.308485:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201555.141234:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201555.332470:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:0.0:1788201556.172091:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201556.358924:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00000100:00100000:3.0:1788201556.487556:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2403:1875062187479296:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201556.487820:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187479552:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201556.487875:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788201556.487932:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187479424:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201556.488030:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187479680:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201556.501585:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187479424:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201556.501609:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2403:1875062187479296:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201556.501623:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187479552:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201556.503415:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187479680:192.168.201.121@tcp:400:kworker.0 00010000:00010000:1.0:1788201557.188848:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201557.381356:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201558.212933:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201558.405019:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:0.0:1788201559.236613:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201559.428470:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00010000:00010000:1.0:1788201560.260627:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201560.452416:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00000100:00080000:3.0:1788201560.556049:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201560.556127:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201560.556131:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 77309411428 gen 12 00000100:00100000:3.0:1788201560.556138:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201560.556145:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 77309411428 last 77309411629 00000100:00080000:3.0:1788201560.556156:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d721880 x1875062187385088/t77309411630(77309411630) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201466 ref 1 fl New:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201560.556169:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187384576:192.168.201.121@tcp:36:mcreate.0 00000100:00100000:3.0:1788201560.556194:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187385088:192.168.201.121@tcp:36:rm.0 00000100:00080000:3.0:1788201560.566177:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d721880 x1875062187385088/t77309411630(77309411630) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201576 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201560.566197:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201560.566200:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 77309411428 gen 12 00000100:00100000:3.0:1788201560.566207:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201560.566217:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 77309411428 last 77309411630 00000100:00080000:3.0:1788201560.566227:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8290fe1500 x1875062187388672/t77309411633(77309411633) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201473 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201560.566240:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187385088:192.168.201.121@tcp:36:rm.0 00000100:00100000:3.0:1788201560.566545:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8290fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187388672:192.168.201.121@tcp:101:createmany.0 00000100:00080000:3.0:1788201560.589836:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8290fe1500 x1875062187388672/t77309411633(77309411633) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201576 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:3.0:1788201560.589866:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e82894d8e00 x1875062187388928/t77309411634(77309411634) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201473 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:3.0:1788201560.589876:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8290fe1500 x1875062187388672/t77309411633(77309411633) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201576 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201560.630164:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201560.630177:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 77309411428 gen 12 00000100:00100000:3.0:1788201560.630199:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201560.630211:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 77309411428 last 77309411633 00000100:00080000:3.0:1788201560.630224:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e82894d8e00 x1875062187388928/t77309411634(77309411634) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201473 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201560.630240:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8290fe1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187388672:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201560.630271:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d8e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187388928:192.168.201.121@tcp:35:createmany.0 00000100:00080000:3.0:1788201560.875076:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8288b93b80 x1875062187397504/t77309411660(77309411660) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201576 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201560.875095:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201560.875099:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 77309411428 gen 12 00000100:00100000:3.0:1788201560.875107:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201560.876419:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 77309411428 last 77309411660 00000100:00080000:3.0:1788201560.876454:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e82894da300 x1875062187397888/t77309411661(77309411661) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201473 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201560.876481:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288b93b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187397504:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201560.876512:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187397888:192.168.201.121@tcp:101:createmany.0 00000100:00080000:3.0:1788201560.882981:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e82894da300 x1875062187397888/t77309411661(77309411661) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201576 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:3.0:1788201560.883016:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e8288bae300 x1875062187398144/t77309411662(77309411662) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201474 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:3.0:1788201560.883026:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e82894da300 x1875062187397888/t77309411661(77309411661) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201576 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201560.883035:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201560.883038:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 77309411428 gen 12 00000100:00100000:3.0:1788201560.883044:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201560.883052:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 77309411428 last 77309411661 00000100:00080000:3.0:1788201560.883065:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8288bae300 x1875062187398144/t77309411662(77309411662) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201474 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201560.883078:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894da300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187397888:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201560.883098:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8288bae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187398144:192.168.201.121@tcp:35:createmany.0 00000100:00080000:3.0:1788201560.886297:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8288bae300 x1875062187398144/t77309411662(77309411662) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201576 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201560.886402:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201560.886405:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 77309411428 gen 12 00000100:00100000:3.0:1788201560.886412:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201560.886422:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 77309411428 last 77309411662 00000100:00080000:3.0:1788201560.886435:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8288baf480 x1875062187398528/t77309411663(77309411663) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201474 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201560.886448:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8288bae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187398144:192.168.201.121@tcp:35:createmany.0 00010000:00010000:0.0:1788201561.288325:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201561.423375:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289550000 x1875062187432448/t77309411769(77309411769) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201577 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:3.0:1788201561.423395:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e8289552a00 x1875062187432704/t77309411770(77309411770) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201475 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:3.0:1788201561.423404:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289550000 x1875062187432448/t77309411769(77309411769) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201577 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201561.423412:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201561.423414:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 77309411428 gen 12 00000100:00100000:3.0:1788201561.423420:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201561.423427:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 77309411428 last 77309411769 00000100:00080000:3.0:1788201561.423452:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289552a00 x1875062187432704/t77309411770(77309411770) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201475 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201561.423464:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289550000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187432448:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201561.423482:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289552a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187432704:192.168.201.121@tcp:35:createmany.0 00000100:00080000:3.0:1788201561.428198:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289552a00 x1875062187432704/t77309411770(77309411770) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201577 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201561.428213:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201561.428216:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 77309411428 gen 12 00000100:00100000:3.0:1788201561.428221:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201561.428230:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 77309411428 last 77309411770 00000100:00080000:3.0:1788201561.428257:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289550380 x1875062187433088/t77309411771(77309411771) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201475 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201561.428269:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289552a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187432704:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201561.428288:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289550380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187433088:192.168.201.121@tcp:101:createmany.0 00000100:00080000:3.0:1788201561.432303:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289550380 x1875062187433088/t77309411771(77309411771) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201577 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:3.0:1788201561.432347:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e8289551c00 x1875062187433344/t77309411772(77309411772) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201475 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:3.0:1788201561.432357:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289550380 x1875062187433088/t77309411771(77309411771) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201577 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201561.432376:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201561.432379:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 77309411428 gen 12 00000100:00100000:3.0:1788201561.432385:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201561.432393:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 77309411428 last 77309411771 00000100:00080000:3.0:1788201561.432419:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289551c00 x1875062187433344/t77309411772(77309411772) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201475 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201561.432435:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289550380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187433088:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201561.432455:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289551c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187433344:192.168.201.121@tcp:35:createmany.0 00000100:00080000:3.0:1788201561.435649:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289551c00 x1875062187433344/t77309411772(77309411772) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201577 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201561.435663:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201561.435666:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 77309411428 gen 12 00000100:00100000:3.0:1788201561.435672:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201561.435679:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 77309411428 last 77309411772 00000100:00080000:3.0:1788201561.435713:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289552d80 x1875062187433728/t77309411773(77309411773) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201475 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201561.435724:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289551c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187433344:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201561.435742:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289552d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187433728:192.168.201.121@tcp:101:createmany.0 00000100:00080000:3.0:1788201561.439573:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289552d80 x1875062187433728/t77309411773(77309411773) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201577 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:3.0:1788201561.439595:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e82894d4380 x1875062187433984/t77309411774(77309411774) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201475 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:3.0:1788201561.439605:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289552d80 x1875062187433728/t77309411773(77309411773) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201577 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201561.439613:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201561.439616:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 77309411428 gen 12 00000100:00100000:3.0:1788201561.439633:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201561.439642:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 77309411428 last 77309411773 00000100:00080000:3.0:1788201561.439667:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e82894d4380 x1875062187433984/t77309411774(77309411774) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201475 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201561.439678:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289552d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187433728:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201561.439703:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187433984:192.168.201.121@tcp:35:createmany.0 00000100:00080000:3.0:1788201561.442249:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e82894d4380 x1875062187433984/t77309411774(77309411774) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201577 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201561.442265:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201561.442268:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 77309411428 gen 12 00000100:00100000:3.0:1788201561.442273:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201561.442279:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 77309411428 last 77309411774 00000100:00080000:3.0:1788201561.442340:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e82894d6a00 x1875062187434368/t77309411775(77309411775) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201475 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201561.442352:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187433984:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201561.442369:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187434368:192.168.201.121@tcp:101:createmany.0 00000100:00080000:3.0:1788201561.445618:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e82894d6a00 x1875062187434368/t77309411775(77309411775) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201577 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:3.0:1788201561.445649:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e82894d4000 x1875062187434624/t77309411776(77309411776) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201475 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:3.0:1788201561.445674:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e82894d6a00 x1875062187434368/t77309411775(77309411775) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201577 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201561.445682:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201561.445685:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 77309411428 gen 12 00000100:00100000:3.0:1788201561.445690:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201561.445704:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 77309411428 last 77309411775 00000100:00080000:3.0:1788201561.445731:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e82894d4000 x1875062187434624/t77309411776(77309411776) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201475 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201561.445743:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187434368:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201561.445762:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187434624:192.168.201.121@tcp:35:createmany.0 00000100:00080000:3.0:1788201561.448599:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e82894d4000 x1875062187434624/t77309411776(77309411776) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201577 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201561.448612:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201561.448615:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 77309411428 gen 12 00000100:00100000:3.0:1788201561.448630:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201561.448637:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 77309411428 last 77309411776 00000100:00080000:3.0:1788201561.448663:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e82894d5880 x1875062187435008/t77309411777(77309411777) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201475 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201561.448675:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187434624:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201561.448703:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187435008:192.168.201.121@tcp:101:createmany.0 00000100:00080000:3.0:1788201561.451881:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e82894d5880 x1875062187435008/t77309411777(77309411777) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201577 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:3.0:1788201561.451902:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e82894d6680 x1875062187435264/t77309411778(77309411778) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201475 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:3.0:1788201561.451911:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e82894d5880 x1875062187435008/t77309411777(77309411777) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201577 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201561.451919:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201561.451921:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 77309411428 gen 12 00000100:00100000:3.0:1788201561.451927:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201561.451934:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 77309411428 last 77309411777 00000100:00080000:3.0:1788201561.451962:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e82894d6680 x1875062187435264/t77309411778(77309411778) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201475 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201561.451973:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187435008:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201561.451993:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187435264:192.168.201.121@tcp:35:createmany.0 00000100:00080000:3.0:1788201561.456182:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e82894d6680 x1875062187435264/t77309411778(77309411778) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201577 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201561.456196:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201561.456199:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 77309411428 gen 12 00000100:00100000:3.0:1788201561.456205:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201561.456213:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 77309411428 last 77309411778 00000100:00080000:3.0:1788201561.456239:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289550700 x1875062187435648/t77309411779(77309411779) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201475 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201561.456251:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187435264:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201561.456269:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289550700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187435648:192.168.201.121@tcp:101:createmany.0 00000100:00080000:3.0:1788201561.460143:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289550700 x1875062187435648/t77309411779(77309411779) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201577 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:3.0:1788201561.460165:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e8289551f80 x1875062187435904/t77309411780(77309411780) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201475 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:3.0:1788201561.460174:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289550700 x1875062187435648/t77309411779(77309411779) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201577 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201561.460182:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201561.460185:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 77309411428 gen 12 00000100:00100000:3.0:1788201561.460190:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201561.460199:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 77309411428 last 77309411779 00000100:00080000:3.0:1788201561.460223:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289551f80 x1875062187435904/t77309411780(77309411780) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201475 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201561.460235:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289550700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187435648:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201561.460254:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289551f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187435904:192.168.201.121@tcp:35:createmany.0 00000100:00080000:3.0:1788201561.465095:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289551f80 x1875062187435904/t77309411780(77309411780) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201577 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201561.465110:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201561.465114:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 77309411428 gen 12 00000100:00100000:3.0:1788201561.465120:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201561.465128:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 77309411428 last 77309411780 00000100:00080000:3.0:1788201561.465154:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289553b80 x1875062187436288/t77309411781(77309411781) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201475 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201561.465167:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289551f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187435904:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201561.465186:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289553b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187436288:192.168.201.121@tcp:101:createmany.0 00000100:00080000:3.0:1788201561.468833:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289553b80 x1875062187436288/t77309411781(77309411781) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201577 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:3.0:1788201561.468857:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e8289550a80 x1875062187436544/t77309411782(77309411782) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201475 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:3.0:1788201561.468867:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289553b80 x1875062187436288/t77309411781(77309411781) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201577 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201561.468875:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201561.468878:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 77309411428 gen 12 00000100:00100000:3.0:1788201561.468884:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201561.468892:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 77309411428 last 77309411781 00000100:00080000:3.0:1788201561.468921:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289550a80 x1875062187436544/t77309411782(77309411782) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201475 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201561.468934:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289553b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187436288:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201561.468953:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289550a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187436544:192.168.201.121@tcp:35:createmany.0 00000100:00080000:3.0:1788201561.473215:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289550a80 x1875062187436544/t77309411782(77309411782) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201577 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201561.473232:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201561.473236:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 77309411428 gen 12 00000100:00100000:3.0:1788201561.473243:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828d722680 x1875062187384576/t77309411629(77309411629) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201578 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201561.473252:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 77309411428 last 77309411782 00000100:00080000:3.0:1788201561.473286:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289553100 x1875062187436928/t77309411783(77309411783) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201475 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201561.473298:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289550a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187436544:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201561.473355:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187436928:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201561.476876:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 0 seconds 00000100:00100000:3.0:1788201561.609380:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187480192:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201561.609415:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788201561.609420:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289528700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187480064:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201561.610146:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2403:1875062187479808:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201561.616756:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187480192:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201561.616787:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289528700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187480064:192.168.201.121@tcp:400:kworker.0 00010000:00080000:0.0:1788201561.659490:0:41518:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff8e8282a4b800 to lustre-MDT0000_UUID@192.168.201.121@tcp 00010000:00010000:0.0:1788201561.659495:0:41518:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff8e8282a4b800 (100) 00000080:00200000:0.0:1788201561.659815:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xd6:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.659853:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1abd lrc: 0/0,0 mode: --/PR res: [0x200006991:0xd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf65b expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.659874:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xd5:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.659886:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1aaf lrc: 0/0,0 mode: --/PR res: [0x200006991:0xd5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf62a expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.659914:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xd4:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.659927:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e1aa1 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xd4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf5f9 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.659941:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xd3:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.659952:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558200/0x9ae3109cee9e1a93 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xd3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf5c8 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.659965:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xd2:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.659977:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560a00/0x9ae3109cee9e1a85 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xd2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf597 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.659993:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xd1:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660154:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560400/0x9ae3109cee9e1a77 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xd1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf566 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660173:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xd0:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660184:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e1a69 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xd0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf535 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660200:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xcf:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660211:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e1a5b lrc: 0/0,0 mode: --/PR res: [0x200006991:0xcf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf504 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660224:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xce:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660233:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1a4d lrc: 0/0,0 mode: --/PR res: [0x200006991:0xce:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf4d3 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660259:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xcd:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660271:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560e00/0x9ae3109cee9e1a3f lrc: 0/0,0 mode: --/PR res: [0x200006991:0xcd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf4a2 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660284:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xcc:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660294:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1a31 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf471 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660306:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xcb:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660364:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e1a23 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xcb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf440 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660379:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xca:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660391:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1a15 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf40f expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660405:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xc9:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660416:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1a07 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xc9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf3de expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660430:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xc8:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660440:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e19f9 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xc8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf3ad expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660452:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xc7:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660461:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e19eb lrc: 0/0,0 mode: --/PR res: [0x200006991:0xc7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf37c expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660485:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xc6:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660496:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e19dd lrc: 0/0,0 mode: --/PR res: [0x200006991:0xc6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf34b expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660509:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xc5:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660519:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e19cf lrc: 0/0,0 mode: --/PR res: [0x200006991:0xc5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf31a expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660531:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xc4:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660540:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e19c1 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xc4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf2e9 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660553:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xc3:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660565:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992a00/0x9ae3109cee9e19b3 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xc3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf2b8 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660578:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xc2:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660589:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9ae00/0x9ae3109cee9e19a5 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xc2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf287 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660602:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xc1:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660612:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e400/0x9ae3109cee9e1997 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf256 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660829:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xc0:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660843:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a000/0x9ae3109cee9e1989 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf225 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660869:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xbf:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660882:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9be00/0x9ae3109cee9e197b lrc: 0/0,0 mode: --/PR res: [0x200006991:0xbf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf1f4 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660894:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xbe:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660904:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9b000/0x9ae3109cee9e196d lrc: 0/0,0 mode: --/PR res: [0x200006991:0xbe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf1c3 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660914:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xbd:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660923:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9bc00/0x9ae3109cee9e195f lrc: 0/0,0 mode: --/PR res: [0x200006991:0xbd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf192 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660936:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xbc:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660946:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a800/0x9ae3109cee9e1951 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xbc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf161 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660959:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xbb:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.660971:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d800/0x9ae3109cee9e1943 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf130 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.660985:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xba:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.661054:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948cc00/0x9ae3109cee9e1935 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xba:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf0ff expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.661071:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xb9:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.661084:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9b200/0x9ae3109cee9e1927 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xb9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf0ce expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.661110:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xb8:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.661121:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a200/0x9ae3109cee9e1919 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf09d expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.661134:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xb7:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.661146:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9b800/0x9ae3109cee9e190b lrc: 0/0,0 mode: --/PR res: [0x200006991:0xb7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf06c expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.661158:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xb6:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.661169:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a600/0x9ae3109cee9e18fd lrc: 0/0,0 mode: --/PR res: [0x200006991:0xb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf03b expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.661182:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xb5:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.661192:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9b600/0x9ae3109cee9e18ef lrc: 0/0,0 mode: --/PR res: [0x200006991:0xb5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbf00a expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.661205:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xb4:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.661215:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e18e1 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xb4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbefd9 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.661230:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xb3:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.661240:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9ba00/0x9ae3109cee9e18d3 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xb3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbefa8 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.661253:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xb2:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.661262:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9ac00/0x9ae3109cee9e18c5 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xb2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbef77 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.661290:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xb1:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.661304:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d400/0x9ae3109cee9e18b7 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xb1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbef46 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.661358:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xb0:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.661371:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9b400/0x9ae3109cee9e18a9 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xb0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbef15 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.661384:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xaf:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.661396:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9aa00/0x9ae3109cee9e189b lrc: 0/0,0 mode: --/PR res: [0x200006991:0xaf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbeee4 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.661410:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xae:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.661420:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8288b9a400/0x9ae3109cee9e188d lrc: 0/0,0 mode: --/PR res: [0x200006991:0xae:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbeeb3 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.661432:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xad:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.661442:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948ee00/0x9ae3109cee9e187f lrc: 0/0,0 mode: --/PR res: [0x200006991:0xad:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbee82 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.661456:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xac:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.661466:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948c800/0x9ae3109cee9e1871 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xac:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbee51 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.661481:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xab:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.661491:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d000/0x9ae3109cee9e1863 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xab:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbee20 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.661516:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xaa:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.661529:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e000/0x9ae3109cee9e1855 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xaa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbedef expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.661544:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xa9:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.661554:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f600/0x9ae3109cee9e1847 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xa9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbedbe expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.661567:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xa8:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.661577:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948fc00/0x9ae3109cee9e1839 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xa8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbed8d expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.661590:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xa7:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.661600:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e182b lrc: 0/0,0 mode: --/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbed5c expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.661612:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xa6:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.661878:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948ea00/0x9ae3109cee9e181d lrc: 0/0,0 mode: --/PR res: [0x200006991:0xa6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbed2b expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.661923:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xa5:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.661935:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e180f lrc: 0/0,0 mode: --/PR res: [0x200006991:0xa5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbecfa expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.661950:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xa4:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.661963:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948ec00/0x9ae3109cee9e1801 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xa4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbecc9 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.661977:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xa3:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.661988:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948c200/0x9ae3109cee9e17f3 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xa3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbec98 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662064:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xa2:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662079:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948fe00/0x9ae3109cee9e17e5 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xa2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbec67 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662094:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xa1:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662104:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948c600/0x9ae3109cee9e17d7 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xa1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbec36 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662131:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xa0:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662144:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e200/0x9ae3109cee9e17c9 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xa0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbec05 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662158:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x9f:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662168:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948c000/0x9ae3109cee9e17bb lrc: 0/0,0 mode: --/PR res: [0x200006991:0x9f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbebd4 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662183:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x9e:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662194:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f000/0x9ae3109cee9e17ad lrc: 0/0,0 mode: --/PR res: [0x200006991:0x9e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbeba3 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662208:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x9d:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662219:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e179f lrc: 0/0,0 mode: --/PR res: [0x200006991:0x9d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbeb72 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662232:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x9c:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662242:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1791 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbeb41 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662256:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x9b:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662266:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e1783 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbeb10 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662280:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x9a:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662292:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f200/0x9ae3109cee9e1775 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbeadf expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662351:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x99:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662364:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948da00/0x9ae3109cee9e1767 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x99:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbeaae expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662378:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x98:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662388:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1759 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x98:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbea7d expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662401:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x97:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662411:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948ce00/0x9ae3109cee9e174b lrc: 0/0,0 mode: --/PR res: [0x200006991:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbea4c expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662425:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x96:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662436:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e800/0x9ae3109cee9e173d lrc: 0/0,0 mode: --/PR res: [0x200006991:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbea1b expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662450:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x95:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662460:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948e600/0x9ae3109cee9e172f lrc: 0/0,0 mode: --/PR res: [0x200006991:0x95:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe9ea expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662473:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x94:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662483:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948d600/0x9ae3109cee9e1721 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe9b9 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662498:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x93:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662508:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f800/0x9ae3109cee9e1713 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x93:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe988 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662534:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x92:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662545:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948f400/0x9ae3109cee9e1705 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe957 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662558:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x91:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662569:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e16f7 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x91:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe926 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662582:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x90:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662592:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948ca00/0x9ae3109cee9e16e9 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x90:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe8f5 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662605:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x8f:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662772:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948c400/0x9ae3109cee9e16db lrc: 0/0,0 mode: --/PR res: [0x200006991:0x8f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe8c4 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662802:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x8e:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662817:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948fa00/0x9ae3109cee9e16cd lrc: 0/0,0 mode: --/PR res: [0x200006991:0x8e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe893 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662831:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x8d:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662842:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e16bf lrc: 0/0,0 mode: --/PR res: [0x200006991:0x8d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe862 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662857:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x8c:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662867:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204a200/0x9ae3109cee9e16b1 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x8c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe831 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662894:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x8b:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662906:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e16a3 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe800 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662920:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x8a:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662932:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948de00/0x9ae3109cee9e1695 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe7cf expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662947:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x89:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662958:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e1687 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x89:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe79e expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.662972:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x88:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.662983:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828948dc00/0x9ae3109cee9e1679 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe76d expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663159:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x87:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663175:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828950d800/0x9ae3109cee9e166b lrc: 0/0,0 mode: --/PR res: [0x200006991:0x87:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe73c expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663190:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x86:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663202:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f99200/0x9ae3109cee9e165d lrc: 0/0,0 mode: --/PR res: [0x200006991:0x86:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe70b expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663216:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x85:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663225:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204a600/0x9ae3109cee9e164f lrc: 0/0,0 mode: --/PR res: [0x200006991:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe6da expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663252:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x84:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663263:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1641 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe6a9 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663276:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x83:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663288:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204b800/0x9ae3109cee9e1633 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe678 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663303:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x82:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663435:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e82c204a800/0x9ae3109cee9e1625 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x82:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe647 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663452:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x81:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663463:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94ac00/0x9ae3109cee9e1617 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x81:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe616 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663478:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x80:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663488:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e1609 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe5e5 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663502:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x7f:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663513:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94ae00/0x9ae3109cee9e15fb lrc: 0/0,0 mode: --/PR res: [0x200006991:0x7f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe5b4 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663526:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x7e:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663537:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e15ed lrc: 0/0,0 mode: --/PR res: [0x200006991:0x7e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe583 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663565:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x7d:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663577:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828950c800/0x9ae3109cee9e15df lrc: 0/0,0 mode: --/PR res: [0x200006991:0x7d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe552 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663591:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x7c:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663601:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b000/0x9ae3109cee9e15ca lrc: 0/0,0 mode: --/PR res: [0x200006991:0x7c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe513 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663675:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x7b:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663745:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a200/0x9ae3109cee9e15bc lrc: 0/0,0 mode: --/PR res: [0x200006991:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe4e2 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663765:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x7a:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663778:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a400/0x9ae3109cee9e15ae lrc: 0/0,0 mode: --/PR res: [0x200006991:0x7a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe4b1 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663802:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x79:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663814:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b600/0x9ae3109cee9e15a0 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x79:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe480 expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663829:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x78:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663840:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e1592 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe44f expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663854:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x77:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663865:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828950ca00/0x9ae3109cee9e1584 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x77:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe41e expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663891:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x76:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663902:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1576 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x76:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe3ed expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663915:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x75:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663925:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e1568 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe3bc expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663938:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x74:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663948:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e155a lrc: 0/0,0 mode: --/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe38b expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.663961:0:41518:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x73:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.663971:0:41518:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e154c lrc: 0/0,0 mode: --/PR res: [0x200006991:0x73:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cbe35a expref: -99 pid: 41142 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201561.663981:0:41518:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff8e8282a4b800 00000100:00080000:1.0:1788201561.664577:0:41518:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:02000000:3.0:1788201561.687878:0:2400:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff8e8282a4b800: Connection restored to 192.168.201.121@tcp (at 192.168.201.121@tcp) 00000100:00080000:3.0:1788201561.687898:0:2400:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000294d2b41): req@ffff8e82910cb100 x1875062187473536/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201561.687936:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187480320:192.168.201.121@tcp:400:ldlm_lock_repla.0 00000100:00100000:1.0:1788201561.688134:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:3.0:1788201561.703132:0:41485:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb100 pname:cluuid:pid:xid:nid:opc:job lfs:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41485:1875062187473536:192.168.201.121@tcp:101:lfs.0 00010000:00010000:3.0:1788201561.703159:0:41485:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.703171:0:41485:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.703184:0:41485:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.703194:0:41485:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.703209:0:41485:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.703217:0:41485:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.703225:0:41485:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201561.703238:0:41485:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910cb100 x1875062187473536/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201577 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201561.703250:0:41485:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.703262:0:41485:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201561.703269:0:41485:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201561.704755:0:41485:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201561.704774:0:41485:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1ae7 00000080:00010000:0.0:1788201561.704782:0:41485:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201561.704785:0:41485:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ae7 from it ffffaa1800c97cb8 00010000:00010000:0.0:1788201561.704799:0:41485:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.704811:0:41485:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201561.705594:0:41485:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8e8292531988), request_mask 0, flags 0, foreign 0 00000080:00200000:0.0:1788201561.705601:0:41485:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201561.705635:0:41485:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201561.705652:0:41485:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201561.705665:0:41485:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.705672:0:41485:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201561.705679:0:41485:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201561.705682:0:41485:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1ae7 00000080:00010000:0.0:1788201561.705687:0:41485:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ae7 from it ffffaa1800c97cb8 00010000:00010000:0.0:1788201561.705690:0:41485:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201561.705698:0:41485:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201561.705819:0:41485:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201561.705828:0:41485:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201561.705835:0:41485:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201561.705843:0:41485:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201561.705861:0:41485:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201561.705866:0:41485:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201561.705869:0:41485:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1ae7 00000080:00010000:0.0:1788201561.705873:0:41485:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ae7 from it ffffaa1800c97bb0 00010000:00010000:0.0:1788201561.705877:0:41485:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201561.705884:0:41485:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:0.0:1788201561.705893:0:41485:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 44 00000080:00200008:0.0:1788201561.705900:0:41485:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8e8292531988), flags 100000 00000080:00200080:0.0:1788201561.705981:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.705995:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa0000 uarg=0000000000000000 00800000:00000004:0.0:1788201561.706002:0:41485:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8e8282a4b800: age 2910, max_age 3020 00000100:00100000:0.0:1788201561.706045:0:41485:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb100 pname:cluuid:pid:xid:nid:opc:job lfs:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41485:1875062187480448:192.168.201.121@tcp:41:lfs.0 00000100:00100000:0.0:1788201561.706122:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201561.720270:0:41485:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7100 pname:cluuid:pid:xid:nid:opc:job lfs:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41485:1875062187480576:192.168.201.121@tcp:41:lfs.0 00800000:00000004:0.0:1788201561.720300:0:41485:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8e8282a4b800: update ffff8e82871ad178 cache blocks 317418/353529 objects 1023742/1024000 00000080:00200080:0.0:1788201561.720465:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720484:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720492:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720496:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720499:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720515:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720519:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720523:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720526:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720530:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720533:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720536:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720539:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720542:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720546:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720549:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720552:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720555:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720558:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720561:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720564:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720567:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720570:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720573:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720576:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720579:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720582:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720585:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720588:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720591:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720603:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720608:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720611:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720627:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720632:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720635:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720639:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720642:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720645:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720649:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720653:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720656:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720660:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720664:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720668:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720670:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720674:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720677:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720681:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720685:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720689:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720692:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720696:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720699:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720703:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720716:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720722:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720725:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720728:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720732:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720735:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720738:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720742:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720745:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720748:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720752:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720755:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720758:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720762:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720766:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720769:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720773:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720777:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720780:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720783:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720795:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720801:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720805:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720809:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720812:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720824:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720829:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720832:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720836:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720839:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720843:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720847:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720850:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720854:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720857:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720860:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720863:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720867:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720870:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720874:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720877:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720880:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720883:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720886:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720890:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720893:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720896:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720900:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720903:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720907:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720916:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720921:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720925:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720929:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720933:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720937:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720940:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720944:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720948:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720951:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720955:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720958:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720962:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720971:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720974:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720978:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720981:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720985:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.720989:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.720993:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00800000:00000080:0.0:1788201561.721001:0:41485:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.721006:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.721012:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00020000:00000004:0.0:1788201561.721020:0:41485:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8e8282a4b800: age 2910, max_age 3020 00000100:00100000:0.0:1788201561.721059:0:41485:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job lfs:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41485:1875062187480704:192.168.201.121@tcp:13:lfs.0 00000100:00100000:0.0:1788201561.721139:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201561.744743:0:41485:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job lfs:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41485:1875062187480704:192.168.201.121@tcp:13:lfs.0 00020000:00000004:0.0:1788201561.744775:0:41485:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8e8282a4b800: update ffff8e82871ab7c0 cache blocks 901369/958279 objects 261609/262144 00000080:00200080:0.0:1788201561.744850:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.744867:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00020000:00000004:0.0:1788201561.744874:0:41485:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8e8282a4b800: age 2910, max_age 3020 00000100:00100000:0.0:1788201561.744908:0:41485:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job lfs:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41485:1875062187480832:192.168.201.121@tcp:13:lfs.0 00000100:00100000:0.0:1788201561.744999:0:41485:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201561.755569:0:41485:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job lfs:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41485:1875062187480832:192.168.201.121@tcp:13:lfs.0 00020000:00000004:0.0:1788201561.755596:0:41485:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8e8282a4b800: update ffff8e82871a9e08 cache blocks 901369/958279 objects 261641/262144 00000080:00200080:0.0:1788201561.755674:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755692:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755700:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755705:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755709:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755713:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755716:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755719:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755723:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755726:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755730:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755734:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755738:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755741:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755761:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755766:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755771:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755774:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755778:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755782:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755795:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755800:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755805:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755812:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755817:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755821:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755825:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755828:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755832:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755836:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755840:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755843:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755847:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755850:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755853:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755856:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755860:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755863:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755867:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755878:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755883:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755886:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755890:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755893:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755896:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755899:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755903:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755906:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755909:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755912:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755915:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755918:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755922:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755926:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.755929:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.755933:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.756497:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.756504:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.756509:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.756512:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.756516:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.756519:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.756523:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.756526:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.756539:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.756543:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.756548:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.756551:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.756556:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.756559:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.756563:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.756567:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.756571:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.756574:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.756579:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.756583:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.756587:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.756590:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.756593:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.756597:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.756601:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.756604:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:0.0:1788201561.756608:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:0.0:1788201561.756712:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8290fa2c00 uarg=0000000000000000 00000080:00200080:1.0:1788201561.756915:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.756925:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200080:1.0:1788201561.756930:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.756946:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200080:1.0:1788201561.756951:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.756956:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200080:1.0:1788201561.756960:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.756971:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200080:1.0:1788201561.756977:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.756980:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200080:1.0:1788201561.756985:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.756989:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200080:1.0:1788201561.756992:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.756996:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200080:1.0:1788201561.756999:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.757002:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200080:1.0:1788201561.757007:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.757011:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200080:1.0:1788201561.757014:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.757018:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200080:1.0:1788201561.757022:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.757025:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200080:1.0:1788201561.757029:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.757032:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200080:1.0:1788201561.757036:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.757040:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200080:1.0:1788201561.757044:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.757048:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200080:1.0:1788201561.757051:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.757055:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200080:1.0:1788201561.757059:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.757062:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200080:1.0:1788201561.757074:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.757078:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200080:1.0:1788201561.757082:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.757086:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200080:1.0:1788201561.757089:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.757093:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200080:1.0:1788201561.757097:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.757100:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200080:1.0:1788201561.757104:0:41485:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffcbf4942d0 00020000:00000080:1.0:1788201561.757108:0:41485:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e82910c2000 uarg=0000000000000000 00000080:00200008:1.0:1788201561.757121:0:41485:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8e8292531988), flags 100000 00000080:00200000:1.0:1788201561.848634:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201561.848664:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201561.848683:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788201561.848769:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:1.0:1788201561.848777:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201561.848795:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201561.848799:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1ae7 00000080:00010000:1.0:1788201561.848807:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ae7 from it ffffaa1802393c10 00010000:00010000:1.0:1788201561.848811:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788201561.848820:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:1.0:1788201561.848842:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201561.848854:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-0, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201561.848859:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-0, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201561.848872:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-0,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201561.848923:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1aee lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201561.848932:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1aee lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201561.848941:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1aee lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201561.848971:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187480960:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201561.849060:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1600 going to sleep for 11 seconds 00000100:00100000:3.0:1788201561.852511:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187480960:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201561.852541:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1aee lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc057a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.852563:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1aee lrc: 4/1,0 mode: --/PR res: [0x200006991:0x73:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc057a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.852571:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1aee lrc: 4/1,0 mode: --/PR res: [0x200006991:0x73:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc057a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.852582:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1aee lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x73:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc057a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.852592:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1aee lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x73:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc057a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.852604:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1aee lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x73:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc057a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.857191:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1aee lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x73:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc057a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.857216:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1aee lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x73:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc057a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201561.857243:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289512300 x1875062187480960/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201577 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201561.857258:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1aee lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x73:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc057a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201561.857344:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1aee lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x73:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc057a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.857359:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/115 (0 0) 00000080:00200000:3.0:1788201561.857366:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201561.857449:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x73:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201561.857459:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201561.857492:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x73:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201561.857551:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x73:0x0](ffff8e8289e221c8): rc = 0 00000080:00200000:3.0:1788201561.857565:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x73:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201561.857572:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x73:0x0](ffff8e8289e221c8) for lock 0x9ae3109cee9e1aee 00000080:00010000:3.0:1788201561.857581:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x73:0x0](ffff8e8289e221c8) 00000080:00010000:3.0:1788201561.857584:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1aee from it ffffaa1802393e00 00010000:00010000:3.0:1788201561.857588:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1aee lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x73:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc057a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.857598:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1aee lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x73:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc057a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201561.857730:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201561.857748:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201561.857761:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788201561.857771:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:3.0:1788201561.857777:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201561.857781:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201561.857797:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1ae7 00000080:00010000:3.0:1788201561.857802:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ae7 from it ffffaa1802393d70 00010000:00010000:3.0:1788201561.857805:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788201561.857813:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:3.0:1788201561.857822:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201561.857829:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-0, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201561.857853:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788201561.857876:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1aee lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x73:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc057a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201561.857883:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x73:0x0]: losing layout lock 00010000:00010000:3.0:1788201561.857912:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1aee lrc: 1/0,0 mode: --/PR res: [0x200006991:0x73:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc057a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201561.857919:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788201561.857925:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201561.857927:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e1aee lrc: 0/0,0 mode: --/PR res: [0x200006991:0x73:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc057a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201561.857942:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ae7 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc03d6 expref: -99 pid: 41485 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000020:00100000:3.0:1788201561.857958:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201561.857965:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289528700 x1875062187481216/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201561.857987:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289528700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187481216:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201561.858106:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c300 going to sleep for 11 seconds 00000100:00100000:0.0:1788201561.868603:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289528700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187481216:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201561.868850:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x73:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201561.868863:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201561.868919:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x73:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201561.868927:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x73:0x0](ffff8e8289e221c8) 00000080:00200000:1.0:1788201561.869168:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201561.869188:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201561.869245:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1af5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201561.869255:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1af5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201561.869267:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1af5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201561.869299:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187481344:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201561.869406:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1240 going to sleep for 11 seconds 00000100:00100000:3.0:1788201561.874019:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289513100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187481344:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201561.874050:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1af5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05a4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.874062:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1af5 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05a4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.874074:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1af5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc05a4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.874085:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1af5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05a4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.874103:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1af5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc05a4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.874110:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1af5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc05a4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.874118:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1af5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc05a4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201561.874131:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289513100 x1875062187481344/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201577 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201561.874144:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1af5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05a4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.874156:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201561.874163:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201561.874189:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201561.874196:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1af5 00000080:00010000:3.0:1788201561.874201:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201561.874204:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1af5 from it ffffaa1802393c10 00010000:00010000:3.0:1788201561.874208:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1af5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc05a4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.874216:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1af5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05a4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201561.874246:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201561.874266:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-1, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201561.874270:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-1, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201561.874298:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-1,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201561.874456:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1afc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.874465:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1afc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.874473:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1afc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201561.874492:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952ad80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187481472:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201561.874602:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201561.882556:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952ad80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187481472:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201561.882581:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1afc lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05c0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.882601:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1afc lrc: 4/1,0 mode: --/PR res: [0x200006991:0x74:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05c0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.882611:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1afc lrc: 4/1,0 mode: --/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05c0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.882632:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1afc lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc05c0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.882643:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1afc lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05c0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.882655:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1afc lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc05c0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.882664:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1afc lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc05c0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.882671:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1afc lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc05c0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201561.882683:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828952ad80 x1875062187481472/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201577 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201561.882694:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1afc lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05c0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201561.882708:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1afc lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05c0 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.882718:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/116 (0 0) 00000080:00200000:3.0:1788201561.882725:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201561.882769:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x74:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201561.882779:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201561.882815:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x74:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201561.882872:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x74:0x0](ffff8e8289c142c8): rc = 0 00000080:00200000:3.0:1788201561.882887:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x74:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201561.882893:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x74:0x0](ffff8e8289c142c8) for lock 0x9ae3109cee9e1afc 00000080:00010000:3.0:1788201561.882902:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x74:0x0](ffff8e8289c142c8) 00000080:00010000:3.0:1788201561.882906:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1afc from it ffffaa1802393e00 00010000:00010000:3.0:1788201561.882910:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1afc lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc05c0 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.882920:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1afc lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc05c0 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201561.882949:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201561.882965:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201561.882974:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1af5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05a4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201561.882985:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1af5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05a4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201561.882992:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201561.883007:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201561.883011:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1af5 00000080:00010000:3.0:1788201561.883014:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1af5 from it ffffaa1802393d70 00010000:00010000:3.0:1788201561.883018:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1af5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc05a4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201561.883026:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1af5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05a4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201561.883039:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201561.883046:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-1, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201561.883070:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1af5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc05a4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201561.883090:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1afc lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc05c0 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201561.883099:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x74:0x0]: losing layout lock 00010000:00010000:3.0:1788201561.883127:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1afc lrc: 1/0,0 mode: --/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc05c0 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201561.883136:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1af5 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc05a4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201561.883141:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201561.883143:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1afc lrc: 0/0,0 mode: --/PR res: [0x200006991:0x74:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc05c0 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201561.883159:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1af5 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc05a4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201561.883174:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201561.883181:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828952ad80 x1875062187481728/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201561.883203:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952ad80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187481728:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201561.883276:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201561.888492:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952ad80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187481728:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201561.888552:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x74:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201561.888565:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201561.888594:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x74:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201561.888602:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x74:0x0](ffff8e8289c142c8) 00000080:00200000:3.0:1788201561.888714:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201561.888734:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201561.888801:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b03 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.888811:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b03 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.888821:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b03 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201561.888839:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187481856:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201561.888942:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201561.899774:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187481856:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201561.899810:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b03 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05ea expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.899823:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b03 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05ea expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.899835:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b03 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc05ea expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.899846:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b03 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05ea expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.899858:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b03 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc05ea expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.899865:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b03 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc05ea expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.899872:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b03 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc05ea expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201561.899884:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289529f80 x1875062187481856/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201577 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201561.899896:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b03 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05ea expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.899910:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201561.899917:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201561.899943:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201561.899950:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1b03 00000080:00010000:3.0:1788201561.899955:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201561.899958:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b03 from it ffffaa1802393c10 00010000:00010000:3.0:1788201561.899961:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b03 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc05ea expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.899975:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b03 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05ea expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201561.900170:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201561.900193:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-2, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201561.900197:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-2, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201561.900219:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-2,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201561.900260:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b0a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.900268:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b0a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.900276:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b0a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201561.900294:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187481984:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201561.900394:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201561.903879:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187481984:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201561.903904:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b0a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0606 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.903924:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b0a lrc: 4/1,0 mode: --/PR res: [0x200006991:0x75:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0606 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.903933:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b0a lrc: 4/1,0 mode: --/PR res: [0x200006991:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0606 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.903944:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b0a lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0606 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.903954:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b0a lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0606 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.903966:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b0a lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0606 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.903974:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b0a lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0606 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.903981:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b0a lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0606 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201561.903994:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289529f80 x1875062187481984/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201577 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201561.904019:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b0a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0606 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201561.904033:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b0a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0606 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.904042:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/117 (0 0) 00000080:00200000:3.0:1788201561.904048:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201561.904091:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x75:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201561.904104:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201561.904131:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x75:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201561.904187:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x75:0x0](ffff8e8289e23a88): rc = 0 00000080:00200000:3.0:1788201561.904203:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x75:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201561.904209:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x75:0x0](ffff8e8289e23a88) for lock 0x9ae3109cee9e1b0a 00000080:00010000:3.0:1788201561.904220:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x75:0x0](ffff8e8289e23a88) 00000080:00010000:3.0:1788201561.904224:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b0a from it ffffaa1802393e00 00010000:00010000:3.0:1788201561.904228:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b0a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc0606 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.904237:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b0a lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc0606 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201561.904264:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201561.904280:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201561.904289:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b03 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05ea expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201561.904299:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b03 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05ea expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201561.904305:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201561.904339:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201561.904343:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1b03 00000080:00010000:3.0:1788201561.904347:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b03 from it ffffaa1802393d70 00010000:00010000:3.0:1788201561.904351:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b03 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc05ea expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201561.904363:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b03 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc05ea expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201561.904371:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201561.904379:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-2, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201561.904403:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b03 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc05ea expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201561.904426:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b0a lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc0606 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201561.904435:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x75:0x0]: losing layout lock 00010000:00010000:3.0:1788201561.904463:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b0a lrc: 1/0,0 mode: --/PR res: [0x200006991:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0606 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201561.904472:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b03 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc05ea expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201561.904478:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201561.904481:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b0a lrc: 0/0,0 mode: --/PR res: [0x200006991:0x75:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0606 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201561.904499:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b03 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc05ea expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201561.904518:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201561.904526:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289529f80 x1875062187482240/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201561.904549:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187482240:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201561.904936:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201561.912512:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187482240:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201561.912571:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x75:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201561.912582:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201561.912605:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x75:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201561.912611:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x75:0x0](ffff8e8289e23a88) 00000080:00200000:3.0:1788201561.912715:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201561.912735:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201561.912797:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b11 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.912807:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b11 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.912817:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b11 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201561.912837:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289528a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187482368:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201561.912906:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201561.918893:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289528a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187482368:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201561.918918:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b11 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0630 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.918930:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b11 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0630 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.918947:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b11 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0630 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.918958:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b11 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0630 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.918971:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b11 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0630 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.918980:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b11 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0630 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.918988:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b11 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0630 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201561.919043:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289528a80 x1875062187482368/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201577 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201561.919059:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b11 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0630 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.919071:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201561.919078:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201561.919103:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201561.919110:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1b11 00000080:00010000:3.0:1788201561.919115:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201561.919118:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b11 from it ffffaa1802393c10 00010000:00010000:3.0:1788201561.919122:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b11 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0630 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.919130:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b11 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0630 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201561.919152:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201561.919169:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-3, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201561.919173:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-3, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201561.919193:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-3,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201561.919230:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b18 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.919237:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b18 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.919246:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b18 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201561.919264:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289528a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187482496:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201561.919469:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201561.923561:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289528a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187482496:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201561.923588:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b18 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc064c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.923608:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b18 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x76:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc064c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.923633:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b18 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x76:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc064c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.923644:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b18 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x76:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc064c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.923654:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b18 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x76:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc064c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.923666:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b18 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x76:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc064c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.923674:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b18 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x76:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc064c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.923681:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b18 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x76:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc064c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201561.923693:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289528a80 x1875062187482496/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201577 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201561.923704:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b18 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x76:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc064c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201561.923717:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b18 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x76:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc064c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.923732:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/118 (0 0) 00000080:00200000:3.0:1788201561.923739:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201561.923778:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x76:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201561.923798:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201561.923823:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x76:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201561.923878:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x76:0x0](ffff8e8289e242c8): rc = 0 00000080:00200000:3.0:1788201561.923892:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x76:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201561.923899:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x76:0x0](ffff8e8289e242c8) for lock 0x9ae3109cee9e1b18 00000080:00010000:3.0:1788201561.923907:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x76:0x0](ffff8e8289e242c8) 00000080:00010000:3.0:1788201561.923911:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b18 from it ffffaa1802393e00 00010000:00010000:3.0:1788201561.923915:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b18 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x76:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc064c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201561.923924:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b18 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x76:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc064c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201561.923952:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201561.923969:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201561.923977:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b11 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0630 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201561.923988:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b11 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0630 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201561.924003:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201561.924008:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201561.924012:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1b11 00000080:00010000:3.0:1788201561.924016:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b11 from it ffffaa1802393d70 00010000:00010000:3.0:1788201561.924019:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b11 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0630 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201561.924027:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b11 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0630 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201561.924035:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201561.924042:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-3, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201561.924068:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b11 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0630 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201561.924091:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b18 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x76:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc064c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201561.924101:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x76:0x0]: losing layout lock 00010000:00010000:3.0:1788201561.924130:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b18 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x76:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc064c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201561.924138:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b11 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0630 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201561.924143:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201561.924145:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b18 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x76:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc064c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201561.924160:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b11 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0630 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201561.924182:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201561.924189:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289528a80 x1875062187482752/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201561.924211:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289528a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187482752:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201561.924271:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201561.927602:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289528a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187482752:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201561.948745:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187483008:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201561.948771:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560c00/0x9ae3109cee9e1b26 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0692 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201561.948802:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560c00/0x9ae3109cee9e1b26 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x77:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0692 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201561.948811:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560c00/0x9ae3109cee9e1b26 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x77:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0692 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201561.948822:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560c00/0x9ae3109cee9e1b26 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x77:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0692 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201561.948833:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560c00/0x9ae3109cee9e1b26 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x77:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0692 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201561.948845:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560c00/0x9ae3109cee9e1b26 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x77:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0692 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201561.948853:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560c00/0x9ae3109cee9e1b26 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x77:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0692 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201561.948860:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560c00/0x9ae3109cee9e1b26 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x77:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0692 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201561.948888:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d7b80 x1875062187483008/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201577 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201561.948901:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560c00/0x9ae3109cee9e1b26 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x77:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0692 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201561.948912:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560c00/0x9ae3109cee9e1b26 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x77:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0692 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201561.948923:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/119 (0 0) 00000080:00200000:1.0:1788201561.948929:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201561.949504:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x77:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201561.949515:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201561.949549:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x77:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201561.949600:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x77:0x0](ffff8e8289e2a1c8): rc = 0 00000080:00200000:1.0:1788201561.949688:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x77:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:1.0:1788201561.949696:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x77:0x0](ffff8e8289e2a1c8) for lock 0x9ae3109cee9e1b26 00000080:00010000:1.0:1788201561.949706:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x77:0x0](ffff8e8289e2a1c8) 00000080:00010000:1.0:1788201561.949710:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b26 from it ffffaa1802393e00 00010000:00010000:1.0:1788201561.949714:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560c00/0x9ae3109cee9e1b26 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x77:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc0692 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201561.949723:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560c00/0x9ae3109cee9e1b26 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x77:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc0692 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.949755:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201561.949772:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201561.949799:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561600/0x9ae3109cee9e1b1f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0676 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201561.949810:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561600/0x9ae3109cee9e1b1f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0676 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.949817:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201561.949823:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201561.949827:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1b1f 00000080:00010000:1.0:1788201561.949831:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b1f from it ffffaa1802393d70 00010000:00010000:1.0:1788201561.949834:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561600/0x9ae3109cee9e1b1f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0676 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201561.949841:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561600/0x9ae3109cee9e1b1f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0676 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201561.949849:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201561.949857:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-4, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201561.949884:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561600/0x9ae3109cee9e1b1f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0676 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201561.949908:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560c00/0x9ae3109cee9e1b26 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x77:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc0692 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201561.949917:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x77:0x0]: losing layout lock 00010000:00010000:1.0:1788201561.949945:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560c00/0x9ae3109cee9e1b26 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x77:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0692 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201561.949952:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561600/0x9ae3109cee9e1b1f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0676 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201561.949963:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201561.949967:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560c00/0x9ae3109cee9e1b26 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x77:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0692 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201561.949983:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561600/0x9ae3109cee9e1b1f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0676 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201561.951400:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201561.951416:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289513480 x1875062187483264/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201561.951443:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187483264:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201561.951524:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1300 going to sleep for 11 seconds 00000100:00100000:0.0:1788201561.961872:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289513480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187483264:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201561.961932:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x77:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201561.961944:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201561.962006:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x77:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201561.962014:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x77:0x0](ffff8e8289e2a1c8) 00000080:00200000:0.0:1788201561.962098:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201561.962116:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201561.962169:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1b2d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.962178:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1b2d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.962187:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1b2d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201561.962206:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187483392:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201561.962395:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201561.967195:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187483392:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201561.967220:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1b2d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc06bc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.967231:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1b2d lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc06bc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.967243:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1b2d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc06bc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.967252:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1b2d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc06bc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.967264:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1b2d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc06bc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.967301:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1b2d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc06bc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.967341:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1b2d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc06bc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201561.967354:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910cbb80 x1875062187483392/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201577 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201561.967366:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1b2d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc06bc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.967378:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201561.967384:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201561.967411:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201561.967416:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1b2d 00000080:00010000:0.0:1788201561.967421:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201561.967424:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b2d from it ffffaa1802393c10 00010000:00010000:0.0:1788201561.967427:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1b2d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc06bc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.967435:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1b2d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc06bc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.967457:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201561.967488:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-5, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201561.967493:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-5, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201561.967518:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-5,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201561.967557:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1b34 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.967564:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1b34 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.967573:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1b34 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201561.967591:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187483520:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201561.967665:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201561.976133:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187483520:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201561.976157:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1b34 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc06d8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.976177:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1b34 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x78:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc06d8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.976185:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1b34 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc06d8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.976196:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1b34 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc06d8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.976206:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1b34 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc06d8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.976219:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1b34 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc06d8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.976239:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1b34 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc06d8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.976246:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1b34 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc06d8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201561.976258:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910cbb80 x1875062187483520/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201577 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201561.976270:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1b34 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc06d8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201561.976297:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1b34 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc06d8 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.976347:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/120 (0 0) 00000080:00200000:0.0:1788201561.976355:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201561.976445:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x78:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201561.976456:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201561.976496:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x78:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201561.976557:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x78:0x0](ffff8e8289c13a88): rc = 0 00000080:00200000:0.0:1788201561.976572:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x78:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201561.976579:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x78:0x0](ffff8e8289c13a88) for lock 0x9ae3109cee9e1b34 00000080:00010000:0.0:1788201561.976588:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x78:0x0](ffff8e8289c13a88) 00000080:00010000:0.0:1788201561.976591:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b34 from it ffffaa1802393e00 00010000:00010000:0.0:1788201561.976595:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1b34 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc06d8 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201561.976629:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1b34 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc06d8 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.976661:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201561.976678:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201561.976930:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1b2d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc06bc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201561.976943:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1b2d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc06bc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.976950:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201561.976955:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201561.976958:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1b2d 00000080:00010000:0.0:1788201561.976963:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b2d from it ffffaa1802393d70 00010000:00010000:0.0:1788201561.976965:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1b2d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc06bc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201561.976972:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1b2d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc06bc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201561.976980:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201561.976986:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-5, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201561.977070:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1b2d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc06bc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201561.977092:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1b34 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc06d8 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201561.977111:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x78:0x0]: losing layout lock 00010000:00010000:0.0:1788201561.977143:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1b34 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc06d8 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201561.977151:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1b2d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc06bc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201561.977157:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201561.977160:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1b34 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x78:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc06d8 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201561.977174:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1b2d lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc06bc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201561.977190:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201561.977198:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82910cbb80 x1875062187483776/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201561.977218:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187483776:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201561.977880:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201561.981799:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187483776:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201561.982026:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x78:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201561.982042:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201561.982074:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x78:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201561.982082:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x78:0x0](ffff8e8289c13a88) 00000080:00200000:1.0:1788201561.982172:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201561.982194:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201561.982240:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1b3b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201561.982246:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1b3b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201561.982253:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e1b3b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201561.982286:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187483904:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201561.982382:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1cc0 going to sleep for 11 seconds 00000080:00200000:3.0:1788201562.009522:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x7a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201562.009536:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201562.009564:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x7a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201562.009573:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x7a:0x0](ffff8e8289e2ba88) 00000080:00200000:3.0:1788201562.009677:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201562.009811:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201562.009867:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b57 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.009877:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b57 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.009888:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b57 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201562.009910:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952b100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187484928:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201562.010143:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.013517:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952b100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187484928:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201562.013545:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b57 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc078e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.013558:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b57 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc078e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.013569:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b57 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc078e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.013582:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b57 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc078e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.013595:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b57 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc078e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.013603:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b57 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc078e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.013677:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b57 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc078e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201562.013698:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828952b100 x1875062187484928/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201562.013712:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b57 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc078e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.013725:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201562.013732:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201562.013758:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201562.013765:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1b57 00000080:00010000:3.0:1788201562.013770:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201562.013772:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b57 from it ffffaa1802393c10 00010000:00010000:3.0:1788201562.013776:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b57 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc078e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.013794:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b57 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc078e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.013820:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201562.013849:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-8, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201562.013854:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-8, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201562.013876:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-8,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201562.013912:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b5e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.013918:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b5e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.013927:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b5e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201562.013957:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952b100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187485056:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201562.014206:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.017111:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952b100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187485056:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201562.017137:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b5e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc07aa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.017163:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b5e lrc: 4/1,0 mode: --/PR res: [0x200006991:0x7b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc07aa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.017171:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b5e lrc: 4/1,0 mode: --/PR res: [0x200006991:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc07aa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.017180:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b5e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc07aa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.017189:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b5e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc07aa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.017201:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b5e lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc07aa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.017208:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b5e lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc07aa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.017216:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b5e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc07aa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201562.017228:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828952b100 x1875062187485056/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201562.017239:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b5e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc07aa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201562.017256:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b5e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc07aa expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.017267:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/123 (0 0) 00000080:00200000:3.0:1788201562.017273:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201562.017445:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x7b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201562.017457:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201562.017484:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x7b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201562.017545:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x7b:0x0](ffff8e8289e25b88): rc = 0 00000080:00200000:3.0:1788201562.017563:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x7b:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201562.017569:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x7b:0x0](ffff8e8289e25b88) for lock 0x9ae3109cee9e1b5e 00000080:00010000:3.0:1788201562.017579:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x7b:0x0](ffff8e8289e25b88) 00000080:00010000:3.0:1788201562.017582:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b5e from it ffffaa1802393e00 00010000:00010000:3.0:1788201562.017586:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b5e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc07aa expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.017595:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b5e lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc07aa expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.017623:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201562.017639:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201562.017649:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b57 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc078e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.017659:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b57 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc078e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.017671:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201562.017675:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201562.017679:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1b57 00000080:00010000:3.0:1788201562.017683:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b57 from it ffffaa1802393d70 00010000:00010000:3.0:1788201562.017686:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b57 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc078e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.017693:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b57 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc078e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201562.017701:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201562.017708:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-8, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201562.017733:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b57 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc078e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201562.017756:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b5e lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc07aa expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.017765:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x7b:0x0]: losing layout lock 00010000:00010000:3.0:1788201562.017804:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b5e lrc: 1/0,0 mode: --/PR res: [0x200006991:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc07aa expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.017812:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b57 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc078e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201562.017819:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201562.017821:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1b5e lrc: 0/0,0 mode: --/PR res: [0x200006991:0x7b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc07aa expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.017837:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b57 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc078e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201562.017855:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201562.017862:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828952b100 x1875062187485312/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201562.017883:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952b100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187485312:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201562.018115:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.021620:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952b100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187485312:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201562.021678:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x7b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201562.021691:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201562.021718:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x7b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201562.021725:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x7b:0x0](ffff8e8289e25b88) 00000080:00200000:3.0:1788201562.021821:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201562.021840:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201562.021895:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b65 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.021905:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b65 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.021915:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b65 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201562.022092:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187485440:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201562.022169:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.028281:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187485440:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201562.028353:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b65 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc07d4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.028366:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b65 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc07d4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.028378:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b65 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc07d4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.028389:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b65 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc07d4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.028401:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b65 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc07d4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.028409:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b65 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc07d4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.028416:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b65 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc07d4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201562.028429:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289529500 x1875062187485440/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201562.028442:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b65 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc07d4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.028459:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201562.028468:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201562.028500:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201562.028507:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1b65 00000080:00010000:3.0:1788201562.028513:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201562.028515:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b65 from it ffffaa1802393c10 00010000:00010000:3.0:1788201562.028519:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b65 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc07d4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.028528:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e1b65 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc07d4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.028552:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201562.028573:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-9, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201562.028578:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-9, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201562.028598:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-9,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201562.028636:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b6c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.028644:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b6c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.028653:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b6c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201562.028672:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187485568:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201562.028740:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.032228:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187485568:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201562.032253:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b6c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc07f0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.032273:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b6c lrc: 4/1,0 mode: --/PR res: [0x200006991:0x7c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc07f0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.032281:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b6c lrc: 4/1,0 mode: --/PR res: [0x200006991:0x7c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc07f0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.032291:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b6c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x7c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc07f0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.032300:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b6c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x7c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc07f0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.032447:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b6c lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x7c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc07f0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.032456:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b6c lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x7c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc07f0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.032462:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b6c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x7c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc07f0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201562.032475:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289529500 x1875062187485568/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201562.032487:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e1b6c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x7c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc07f0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1788201562.059342:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201562.059350:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1b81 00000080:00010000:0.0:1788201562.059356:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.059359:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b81 from it ffffaa1802393c10 00010000:00010000:0.0:1788201562.059363:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561e00/0x9ae3109cee9e1b81 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0860 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.059387:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561e00/0x9ae3109cee9e1b81 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0860 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.059411:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201562.059433:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-11, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201562.059437:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-11, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201562.059456:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-11,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201562.059490:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b88 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.059495:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b88 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.059501:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b88 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.059516:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c8e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187486592:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.059647:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.065079:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c8e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187486592:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.065108:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b88 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc087c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.065128:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b88 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x7e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc087c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.065138:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b88 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x7e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc087c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.065163:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b88 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x7e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc087c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.065174:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b88 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x7e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc087c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.065186:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b88 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x7e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc087c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.065194:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b88 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x7e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc087c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.065201:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b88 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x7e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc087c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.065212:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910c8e00 x1875062187486592/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.065223:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b88 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x7e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc087c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201562.065235:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b88 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x7e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc087c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.065244:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/126 (0 0) 00000080:00200000:0.0:1788201562.065250:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201562.065289:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x7e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.065338:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.065368:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x7e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.065426:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x7e:0x0](ffff8e8289e24b08): rc = 0 00000080:00200000:0.0:1788201562.065441:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x7e:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201562.065447:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x7e:0x0](ffff8e8289e24b08) for lock 0x9ae3109cee9e1b88 00000080:00010000:0.0:1788201562.065457:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x7e:0x0](ffff8e8289e24b08) 00000080:00010000:0.0:1788201562.065461:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b88 from it ffffaa1802393e00 00010000:00010000:0.0:1788201562.065463:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b88 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x7e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc087c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.065470:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b88 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x7e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc087c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.065497:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.065509:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.065515:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561e00/0x9ae3109cee9e1b81 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0860 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.065523:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561e00/0x9ae3109cee9e1b81 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0860 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.065527:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201562.065529:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.065531:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1b81 00000080:00010000:0.0:1788201562.065533:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b81 from it ffffaa1802393d70 00010000:00010000:0.0:1788201562.065535:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561e00/0x9ae3109cee9e1b81 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0860 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.065550:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561e00/0x9ae3109cee9e1b81 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0860 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201562.065559:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201562.065566:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-11, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201562.065589:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561e00/0x9ae3109cee9e1b81 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0860 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.065611:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b88 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x7e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc087c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.065620:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x7e:0x0]: losing layout lock 00010000:00010000:0.0:1788201562.065820:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b88 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x7e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc087c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.065830:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561e00/0x9ae3109cee9e1b81 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0860 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.065836:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201562.065839:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b88 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x7e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc087c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.065854:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561e00/0x9ae3109cee9e1b81 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0860 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201562.065870:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201562.065876:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82910c8e00 x1875062187486848/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201562.065898:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c8e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187486848:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201562.065958:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.069142:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c8e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187486848:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201562.069204:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x7e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.069218:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.069246:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x7e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.069253:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x7e:0x0](ffff8e8289e24b08) 00000080:00200000:0.0:1788201562.069375:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.069396:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.069451:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1b8f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.069459:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1b8f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.069466:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1b8f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.069480:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187486976:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.069537:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.081222:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187486976:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.081244:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1b8f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08a6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.081254:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1b8f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08a6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.081286:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1b8f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc08a6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.081293:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1b8f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08a6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.081303:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1b8f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc08a6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.081406:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1b8f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc08a6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.081411:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1b8f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc08a6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.081419:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910cb480 x1875062187486976/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.081428:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1b8f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08a6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.081438:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201562.081443:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201562.081468:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201562.081472:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1b8f 00000080:00010000:0.0:1788201562.081476:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.081478:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b8f from it ffffaa1802393c10 00010000:00010000:0.0:1788201562.081485:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1b8f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc08a6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.081491:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1b8f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08a6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.081511:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201562.081530:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-12, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201562.081534:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-12, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201562.081604:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-12,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201562.081684:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b96 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.081693:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b96 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.081701:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b96 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.081720:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187487104:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.081824:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.083877:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187487104:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.083900:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b96 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08c2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.083916:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b96 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x7f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08c2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.083937:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b96 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x7f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08c2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.083949:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b96 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x7f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc08c2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.083959:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b96 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x7f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08c2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.083970:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b96 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x7f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc08c2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.083978:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b96 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x7f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc08c2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.083986:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b96 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x7f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc08c2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.084040:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910cb480 x1875062187487104/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.084049:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b96 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x7f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08c2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201562.084058:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b96 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x7f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08c2 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.084074:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/127 (0 0) 00000080:00200000:0.0:1788201562.084079:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201562.084112:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x7f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.084123:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.084150:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x7f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.084215:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x7f:0x0](ffff8e8289e34b08): rc = 0 00000080:00200000:0.0:1788201562.084230:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x7f:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201562.084235:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x7f:0x0](ffff8e8289e34b08) for lock 0x9ae3109cee9e1b96 00000080:00010000:0.0:1788201562.084245:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x7f:0x0](ffff8e8289e34b08) 00000080:00010000:0.0:1788201562.084249:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b96 from it ffffaa1802393e00 00010000:00010000:0.0:1788201562.084252:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b96 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x7f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc08c2 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.084261:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b96 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x7f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc08c2 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.084288:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.084303:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.084368:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1b8f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08a6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.084379:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1b8f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08a6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.084386:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201562.084391:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.084399:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1b8f 00000080:00010000:0.0:1788201562.084404:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b8f from it ffffaa1802393d70 00010000:00010000:0.0:1788201562.084407:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1b8f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc08a6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.084415:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1b8f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08a6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201562.084424:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201562.084431:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-12, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201562.084452:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1b8f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc08a6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.084468:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b96 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x7f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc08c2 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.084474:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x7f:0x0]: losing layout lock 00010000:00010000:0.0:1788201562.084498:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b96 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x7f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc08c2 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.084502:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1b8f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc08a6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.084506:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201562.084508:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1b96 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x7f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc08c2 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.084518:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1b8f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc08a6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201562.084540:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201562.084547:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82910cb480 x1875062187487360/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201562.084568:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187487360:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201562.084634:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.087061:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187487360:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201562.087119:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x7f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.087129:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.087152:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x7f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.087157:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x7f:0x0](ffff8e8289e34b08) 00000080:00200000:0.0:1788201562.087229:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.087244:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.087297:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1b9d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.087305:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1b9d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.087415:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1b9d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.087430:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c8a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187487488:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.087512:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.089268:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c8a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187487488:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.089393:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1b9d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08ec expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.089402:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1b9d lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08ec expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.089410:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1b9d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc08ec expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.089417:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1b9d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08ec expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.089426:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1b9d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc08ec expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.089431:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1b9d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc08ec expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.089435:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1b9d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc08ec expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.089443:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910c8a80 x1875062187487488/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.089453:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1b9d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08ec expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.089461:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201562.089465:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201562.089486:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201562.089490:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1b9d 00000080:00010000:0.0:1788201562.089494:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.089496:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b9d from it ffffaa1802393c10 00010000:00010000:0.0:1788201562.089499:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1b9d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc08ec expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.089505:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1b9d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08ec expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.089522:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201562.089553:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-13, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201562.089556:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-13, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201562.089572:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-13,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201562.089597:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ba4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.089602:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ba4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.089614:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ba4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.089630:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c8a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187487616:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.089732:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.091718:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c8a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187487616:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.091736:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ba4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0908 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.091753:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ba4 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x80:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0908 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.091758:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ba4 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0908 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.091765:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ba4 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0908 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.091772:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ba4 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0908 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.091780:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ba4 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0908 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.091798:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ba4 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0908 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.091805:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ba4 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0908 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.091817:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910c8a80 x1875062187487616/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.091827:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ba4 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0908 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201562.091845:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ba4 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0908 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.091853:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/128 (0 0) 00000080:00200000:0.0:1788201562.091857:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201562.091891:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x80:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.091898:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.091921:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x80:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.091975:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x80:0x0](ffff8e8289c0c2c8): rc = 0 00000080:00200000:0.0:1788201562.091985:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x80:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201562.091989:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x80:0x0](ffff8e8289c0c2c8) for lock 0x9ae3109cee9e1ba4 00000080:00010000:0.0:1788201562.091997:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x80:0x0](ffff8e8289c0c2c8) 00000080:00010000:0.0:1788201562.092000:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ba4 from it ffffaa1802393e00 00010000:00010000:0.0:1788201562.092004:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ba4 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc0908 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.092012:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ba4 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc0908 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.092486:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.092507:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.092517:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1b9d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08ec expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.092526:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1b9d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08ec expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.092534:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201562.092537:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.092540:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1b9d 00000080:00010000:0.0:1788201562.092799:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1b9d from it ffffaa1802393d70 00010000:00010000:0.0:1788201562.092805:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1b9d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc08ec expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.092813:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1b9d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc08ec expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201562.092823:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201562.092830:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-13, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201562.092859:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1b9d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc08ec expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.092877:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ba4 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc0908 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.092883:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x80:0x0]: losing layout lock 00010000:00010000:0.0:1788201562.092912:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ba4 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0908 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.092917:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1b9d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc08ec expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.092920:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201562.092922:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1ba4 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x80:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0908 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.092945:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1b9d lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc08ec expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201562.092958:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201562.092965:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82910c8a80 x1875062187487872/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201562.092981:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c8a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187487872:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201562.093102:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.096261:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c8a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187487872:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201562.096510:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x80:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.096521:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.096553:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x80:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.096559:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x80:0x0](ffff8e8289c0c2c8) 00000080:00200000:0.0:1788201562.096640:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.096799:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.096857:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1bab lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.096867:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1bab lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.096876:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1bab lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.096895:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187488000:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.097133:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.103532:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187488000:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.103603:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1bab lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0932 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.103648:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1bab lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0932 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.103659:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1bab lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0932 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.103671:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1bab lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0932 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.103732:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1bab lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0932 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.103739:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1bab lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0932 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.103743:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1bab lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0932 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.103750:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910c9500 x1875062187488000/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.103760:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1bab lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0932 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.103773:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201562.103778:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201562.103833:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201562.103838:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1bab 00000080:00010000:0.0:1788201562.103842:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.103844:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1bab from it ffffaa1802393c10 00010000:00010000:0.0:1788201562.103847:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1bab lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0932 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.103853:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1bab lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0932 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.103873:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201562.103891:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-14, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201562.103895:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-14, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201562.103914:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-14,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201562.104003:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1bb2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.104011:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1bb2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.104021:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1bb2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.104042:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187488128:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.104165:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.106905:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187488128:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.106985:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1bb2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc094e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.107005:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1bb2 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x81:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc094e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.107013:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1bb2 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x81:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc094e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.107026:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1bb2 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x81:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc094e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.107037:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1bb2 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x81:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc094e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.107050:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1bb2 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x81:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc094e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.107057:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1bb2 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x81:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc094e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.107064:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1bb2 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x81:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc094e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.107075:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910c9500 x1875062187488128/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.107084:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1bb2 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x81:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc094e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201562.107098:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1bb2 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x81:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc094e expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.107107:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/129 (0 0) 00000080:00200000:0.0:1788201562.107113:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201562.107155:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x81:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.107166:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.107193:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x81:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.107258:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x81:0x0](ffff8e8289e300c8): rc = 0 00000080:00200000:0.0:1788201562.107461:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x81:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201562.107466:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x81:0x0](ffff8e8289e300c8) for lock 0x9ae3109cee9e1bb2 00000080:00010000:0.0:1788201562.107476:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x81:0x0](ffff8e8289e300c8) 00000080:00010000:0.0:1788201562.107478:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1bb2 from it ffffaa1802393e00 00010000:00010000:0.0:1788201562.107480:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1bb2 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x81:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc094e expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.107487:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1bb2 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x81:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc094e expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.107512:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.107524:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.107531:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1bab lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0932 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.107538:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1bab lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0932 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.107555:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201562.107558:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.107560:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1bab 00000080:00010000:0.0:1788201562.107562:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1bab from it ffffaa1802393d70 00010000:00010000:0.0:1788201562.107564:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1bab lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0932 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.107569:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1bab lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0932 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201562.107574:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201562.107579:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-14, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201562.107598:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1bab lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0932 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.107623:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1bb2 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x81:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc094e expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.107629:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x81:0x0]: losing layout lock 00010000:00010000:0.0:1788201562.107654:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1bb2 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x81:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc094e expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.107662:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1bab lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0932 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.107668:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201562.107671:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1bb2 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x81:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc094e expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.107695:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1bab lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0932 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201562.107714:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201562.107721:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82910c9500 x1875062187488384/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201562.107745:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187488384:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201562.107864:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.110695:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187488384:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201562.110746:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x81:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.110756:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.110789:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x81:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.110797:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x81:0x0](ffff8e8289e300c8) 00000080:00200000:0.0:1788201562.110887:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.110909:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.111012:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bb9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.111021:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bb9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.111031:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bb9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.111051:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187488512:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.111162:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.113108:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187488512:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.113128:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bb9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0978 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.113136:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bb9 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0978 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.113143:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bb9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0978 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.113151:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bb9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0978 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.113163:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bb9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0978 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.113170:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bb9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0978 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.113177:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bb9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0978 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.113189:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910cb800 x1875062187488512/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.113202:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bb9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0978 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.113216:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201562.113221:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201562.113241:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201562.113245:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1bb9 00000080:00010000:0.0:1788201562.113249:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.113251:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1bb9 from it ffffaa1802393c10 00010000:00010000:0.0:1788201562.113253:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bb9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0978 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.113259:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bb9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0978 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.113291:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201562.113332:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-15, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201562.113336:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-15, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201562.113353:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-15,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201562.113383:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bc0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.113387:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bc0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.113393:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bc0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.113406:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187488640:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.113482:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.115478:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187488640:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.115496:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bc0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0994 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.115511:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bc0 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x82:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0994 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.115516:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bc0 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x82:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0994 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.115523:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bc0 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x82:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0994 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.115529:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bc0 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x82:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0994 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.115536:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bc0 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x82:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0994 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.115541:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bc0 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x82:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0994 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.115548:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bc0 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x82:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0994 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.115558:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910cb800 x1875062187488640/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.115568:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bc0 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x82:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0994 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201562.115581:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bc0 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x82:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0994 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.115589:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/130 (0 0) 00000080:00200000:0.0:1788201562.115596:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201562.115670:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x82:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.115679:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.115710:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x82:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.115765:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x82:0x0](ffff8e8289e32a08): rc = 0 00000080:00200000:0.0:1788201562.115776:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x82:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201562.115790:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x82:0x0](ffff8e8289e32a08) for lock 0x9ae3109cee9e1bc0 00000080:00010000:0.0:1788201562.115800:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x82:0x0](ffff8e8289e32a08) 00000080:00010000:0.0:1788201562.115802:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1bc0 from it ffffaa1802393e00 00010000:00010000:0.0:1788201562.115805:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bc0 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x82:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc0994 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.115811:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bc0 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x82:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc0994 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.115832:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.115844:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.115850:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bb9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0978 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.115858:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bb9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0978 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.115865:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201562.115868:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.115870:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1bb9 00000080:00010000:0.0:1788201562.115872:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1bb9 from it ffffaa1802393d70 00010000:00010000:0.0:1788201562.115874:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bb9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0978 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.115878:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bb9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0978 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201562.115882:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201562.115888:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-15, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201562.115905:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bb9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0978 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.115920:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bc0 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x82:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc0994 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.115925:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x82:0x0]: losing layout lock 00010000:00010000:0.0:1788201562.116007:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bc0 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x82:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0994 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.116016:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bb9 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0978 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.116021:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201562.116024:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bc0 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x82:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0994 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.116040:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bb9 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0978 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201562.116058:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201562.116066:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82910cb800 x1875062187488896/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201562.116086:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187488896:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201562.116183:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.118609:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187488896:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201562.118663:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x82:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.118672:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.118693:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x82:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.118698:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x82:0x0](ffff8e8289e32a08) 00000080:00200000:0.0:1788201562.118774:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.118802:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.118845:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bc7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.118851:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bc7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.118858:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bc7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.118870:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910ca300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187489024:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.118960:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.121296:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910ca300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187489024:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.121340:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bc7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc09be expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.121350:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bc7 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc09be expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.121357:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bc7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc09be expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.121364:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bc7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc09be expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.121373:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bc7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc09be expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.121378:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bc7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc09be expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.121382:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bc7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc09be expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.121390:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910ca300 x1875062187489024/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.121399:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bc7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc09be expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.121411:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201562.121416:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201562.121437:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201562.121441:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1bc7 00000080:00010000:0.0:1788201562.121444:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.121446:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1bc7 from it ffffaa1802393c10 00010000:00010000:0.0:1788201562.121449:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bc7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc09be expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.121454:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bc7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc09be expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.121470:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201562.121488:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-16, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201562.121492:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-16, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201562.121506:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-16,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201562.121532:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bce lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.121536:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bce lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.121542:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bce lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.121558:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910ca300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187489152:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.121625:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.126886:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910ca300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187489152:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.126916:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bce lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc09da expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.126938:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bce lrc: 4/1,0 mode: --/PR res: [0x200006991:0x83:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc09da expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.126948:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bce lrc: 4/1,0 mode: --/PR res: [0x200006991:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc09da expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.126960:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bce lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc09da expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.126972:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bce lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc09da expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.126985:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bce lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc09da expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.126994:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bce lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc09da expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.127001:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bce lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc09da expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.127014:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910ca300 x1875062187489152/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.127025:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bce lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc09da expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201562.127039:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bce lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc09da expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.127050:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/131 (0 0) 00000080:00200000:0.0:1788201562.127057:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201562.127098:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x83:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.127110:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.127138:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x83:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.127200:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x83:0x0](ffff8e8289e33248): rc = 0 00000080:00200000:0.0:1788201562.127215:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x83:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201562.127221:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x83:0x0](ffff8e8289e33248) for lock 0x9ae3109cee9e1bce 00000080:00010000:0.0:1788201562.127231:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x83:0x0](ffff8e8289e33248) 00000080:00010000:0.0:1788201562.127234:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1bce from it ffffaa1802393e00 00010000:00010000:0.0:1788201562.127238:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bce lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc09da expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.127246:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bce lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc09da expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.127287:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.127304:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.127353:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bc7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc09be expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.127363:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bc7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc09be expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.127375:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201562.127379:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.127381:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1bc7 00000080:00010000:0.0:1788201562.127384:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1bc7 from it ffffaa1802393d70 00010000:00010000:0.0:1788201562.127386:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bc7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc09be expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.127391:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bc7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc09be expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201562.127396:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201562.127402:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-16, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201562.127421:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bc7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc09be expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.127436:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bce lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc09da expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.127441:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x83:0x0]: losing layout lock 00010000:00010000:0.0:1788201562.127464:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bce lrc: 1/0,0 mode: --/PR res: [0x200006991:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc09da expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.127469:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bc7 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc09be expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.127472:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201562.127474:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bce lrc: 0/0,0 mode: --/PR res: [0x200006991:0x83:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc09da expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.127486:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bc7 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc09be expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201562.127497:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201562.127502:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82910ca300 x1875062187489408/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201562.127518:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910ca300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187489408:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201562.127577:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.138105:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910ca300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187489408:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201562.138172:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x83:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.138186:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.138217:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x83:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.138225:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x83:0x0](ffff8e8289e33248) 00000080:00200000:0.0:1788201562.139135:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.139163:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.139229:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bd5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.139240:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bd5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.139250:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bd5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.139283:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187489536:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.139388:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.144182:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187489536:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.144211:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bd5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a04 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.144223:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bd5 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a04 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.144233:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bd5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a04 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.144244:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bd5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a04 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.144256:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bd5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a04 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.144265:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bd5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a04 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.144272:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bd5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a04 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.144284:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910cb100 x1875062187489536/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.144297:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bd5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a04 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.144345:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201562.144352:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201562.144382:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201562.144388:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1bd5 00000080:00010000:0.0:1788201562.144393:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.144396:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1bd5 from it ffffaa1802393c10 00010000:00010000:0.0:1788201562.144399:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bd5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a04 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.144408:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bd5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a04 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.144432:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201562.144452:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-17, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201562.144457:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-17, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201562.144477:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-17,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201562.144518:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bdc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.144525:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bdc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.144535:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bdc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.144555:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187489664:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.144635:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.150489:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187489664:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.150519:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bdc lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a20 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.150539:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bdc lrc: 4/1,0 mode: --/PR res: [0x200006991:0x84:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a20 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.150547:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bdc lrc: 4/1,0 mode: --/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a20 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.150558:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bdc lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a20 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.150569:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bdc lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a20 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.150582:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bdc lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a20 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.150590:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bdc lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a20 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.150598:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bdc lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a20 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.150621:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910cb100 x1875062187489664/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.150634:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bdc lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a20 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201562.150648:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bdc lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a20 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.150659:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/132 (0 0) 00000080:00200000:0.0:1788201562.150666:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201562.150709:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x84:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.150720:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.150747:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x84:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.150820:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x84:0x0](ffff8e8289c163c8): rc = 0 00000080:00200000:0.0:1788201562.150835:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x84:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201562.150842:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x84:0x0](ffff8e8289c163c8) for lock 0x9ae3109cee9e1bdc 00000080:00010000:0.0:1788201562.150853:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x84:0x0](ffff8e8289c163c8) 00000080:00010000:0.0:1788201562.150856:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1bdc from it ffffaa1802393e00 00010000:00010000:0.0:1788201562.150860:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bdc lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc0a20 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.150868:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bdc lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc0a20 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.150901:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.150917:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.150927:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bd5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a04 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.150938:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bd5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a04 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.150950:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201562.150954:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.150957:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1bd5 00000080:00010000:0.0:1788201562.150961:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1bd5 from it ffffaa1802393d70 00010000:00010000:0.0:1788201562.150965:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bd5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a04 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.150971:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bd5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a04 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201562.150979:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201562.150986:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-17, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201562.151011:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bd5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0a04 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.151033:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bdc lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc0a20 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.151042:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x84:0x0]: losing layout lock 00010000:00010000:0.0:1788201562.151074:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bdc lrc: 1/0,0 mode: --/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0a20 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.151083:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bd5 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0a04 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.151088:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201562.151092:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bdc lrc: 0/0,0 mode: --/PR res: [0x200006991:0x84:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0a20 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.151105:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bd5 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0a04 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201562.151123:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201562.151130:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82910cb100 x1875062187489920/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201562.151151:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187489920:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201562.151234:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.158813:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187489920:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201562.158876:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x84:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.158889:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.158918:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x84:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.158938:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x84:0x0](ffff8e8289c163c8) 00000080:00200000:0.0:1788201562.159042:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.159061:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.159116:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1be3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.159125:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1be3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.159135:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1be3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.159153:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187490048:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.159228:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.163714:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187490048:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.163742:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1be3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a4a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.163754:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1be3 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a4a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.163765:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1be3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a4a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.163776:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1be3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a4a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.163798:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1be3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a4a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.163805:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1be3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a4a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.163811:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1be3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a4a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.163822:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793df80 x1875062187490048/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.163833:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1be3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a4a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.163848:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201562.163854:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201562.163877:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201562.163883:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1be3 00000080:00010000:0.0:1788201562.163887:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.163890:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1be3 from it ffffaa1802393c10 00010000:00010000:0.0:1788201562.163893:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1be3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a4a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.163900:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1be3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a4a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.163922:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201562.163954:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-18, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201562.163959:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-18, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201562.163980:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-18,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201562.164021:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bea lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.164029:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bea lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.164039:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bea lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.164058:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187490176:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.164121:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.171630:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187490176:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.171659:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bea lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a66 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.171793:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bea lrc: 4/1,0 mode: --/PR res: [0x200006991:0x85:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a66 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.171803:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bea lrc: 4/1,0 mode: --/PR res: [0x200006991:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a66 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.171814:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bea lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a66 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.171825:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bea lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a66 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.171839:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bea lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a66 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.171848:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bea lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a66 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.171856:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bea lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a66 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.171868:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793df80 x1875062187490176/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.171879:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bea lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a66 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201562.171895:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bea lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a66 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.171905:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/133 (0 0) 00000080:00200000:0.0:1788201562.171913:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201562.171959:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x85:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.171970:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.172262:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x85:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.172387:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x85:0x0](ffff8e8289e31148): rc = 0 00000080:00200000:0.0:1788201562.172404:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x85:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201562.172409:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x85:0x0](ffff8e8289e31148) for lock 0x9ae3109cee9e1bea 00000080:00010000:0.0:1788201562.172420:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x85:0x0](ffff8e8289e31148) 00000080:00010000:0.0:1788201562.172423:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1bea from it ffffaa1802393e00 00010000:00010000:0.0:1788201562.172426:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bea lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc0a66 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.172435:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bea lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc0a66 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.172466:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.172483:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.172493:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1be3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a4a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.172503:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1be3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a4a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.172516:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201562.172520:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.172524:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1be3 00000080:00010000:0.0:1788201562.172528:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1be3 from it ffffaa1802393d70 00010000:00010000:0.0:1788201562.172531:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1be3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a4a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.172539:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1be3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a4a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201562.172777:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201562.172796:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-18, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201562.172824:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1be3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0a4a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.172845:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bea lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc0a66 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.172854:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x85:0x0]: losing layout lock 00010000:00010000:0.0:1788201562.172883:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bea lrc: 1/0,0 mode: --/PR res: [0x200006991:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0a66 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.172890:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1be3 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0a4a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.172896:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201562.172899:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1bea lrc: 0/0,0 mode: --/PR res: [0x200006991:0x85:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0a66 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.172915:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1be3 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0a4a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201562.172932:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201562.172941:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793df80 x1875062187490432/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201562.172963:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187490432:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201562.173302:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.180280:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187490432:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201562.181200:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x85:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.181215:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.181243:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x85:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.181250:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x85:0x0](ffff8e8289e31148) 00000080:00200000:0.0:1788201562.181748:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.181772:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.181839:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bf1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.181850:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bf1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.181859:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bf1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.181881:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ed80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187490560:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.182278:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.187120:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ed80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187490560:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.187213:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bf1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a90 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.187228:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bf1 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a90 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.187240:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bf1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a90 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.187252:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bf1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a90 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.187348:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bf1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a90 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.187358:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bf1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a90 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.187365:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bf1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a90 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.187378:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793ed80 x1875062187490560/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.187391:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bf1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a90 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.187408:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201562.187414:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201562.187441:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201562.187446:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1bf1 00000080:00010000:0.0:1788201562.187452:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.187455:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1bf1 from it ffffaa1802393c10 00010000:00010000:0.0:1788201562.187458:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bf1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a90 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.187467:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bf1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a90 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.187490:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201562.187511:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-19, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201562.187516:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-19, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201562.187535:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-19,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201562.187761:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bf8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.187768:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bf8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.187778:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bf8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.187811:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ed80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187490688:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.187943:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.191844:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ed80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187490688:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.191872:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bf8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0aac expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.191894:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bf8 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x86:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0aac expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.191902:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bf8 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x86:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0aac expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.191914:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bf8 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x86:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0aac expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.191923:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bf8 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x86:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0aac expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.192093:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bf8 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x86:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0aac expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.192101:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bf8 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x86:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0aac expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.192109:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bf8 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x86:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0aac expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.192121:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793ed80 x1875062187490688/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.192132:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bf8 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x86:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0aac expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201562.192213:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bf8 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x86:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0aac expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.192227:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/134 (0 0) 00000080:00200000:0.0:1788201562.192235:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201562.192402:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x86:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.192414:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.192439:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x86:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.192493:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x86:0x0](ffff8e8289e36c08): rc = 0 00000080:00200000:0.0:1788201562.192507:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x86:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201562.192513:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x86:0x0](ffff8e8289e36c08) for lock 0x9ae3109cee9e1bf8 00000080:00010000:0.0:1788201562.192523:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x86:0x0](ffff8e8289e36c08) 00000080:00010000:0.0:1788201562.192526:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1bf8 from it ffffaa1802393e00 00010000:00010000:0.0:1788201562.192529:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bf8 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x86:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc0aac expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.192539:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bf8 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x86:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc0aac expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.192755:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.192774:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.192793:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bf1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a90 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.192804:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bf1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a90 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.192816:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201562.192821:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.192824:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1bf1 00000080:00010000:0.0:1788201562.192827:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1bf1 from it ffffaa1802393d70 00010000:00010000:0.0:1788201562.192831:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bf1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0a90 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.192838:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bf1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0a90 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201562.192847:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201562.192854:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-19, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201562.192880:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bf1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0a90 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.192902:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bf8 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x86:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc0aac expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.192910:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x86:0x0]: losing layout lock 00010000:00010000:0.0:1788201562.193854:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bf8 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x86:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0aac expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.193866:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bf1 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0a90 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.193871:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201562.193874:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0000/0x9ae3109cee9e1bf8 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x86:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0aac expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.193891:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1bf1 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0a90 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201562.193911:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201562.193923:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793ed80 x1875062187490944/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201562.194419:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ed80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187490944:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201562.194554:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.200412:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ed80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187490944:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201562.200474:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x86:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.200487:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.200515:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x86:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.200523:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x86:0x0](ffff8e8289e36c08) 00000080:00200000:0.0:1788201562.200822:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.200842:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.200898:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bff lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.200909:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bff lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.200919:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bff lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.201112:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187491072:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.201415:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.204917:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187491072:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.205118:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bff lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ad6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.205130:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bff lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ad6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.205141:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bff lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ad6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.205216:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bff lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ad6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.205232:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bff lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ad6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.205240:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bff lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ad6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.205247:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bff lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ad6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.205342:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793d500 x1875062187491072/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.205361:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ad6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.205379:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201562.205386:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201562.205411:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201562.205417:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1bff 00000080:00010000:0.0:1788201562.205423:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.205426:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1bff from it ffffaa1802393c10 00010000:00010000:0.0:1788201562.205430:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ad6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.205438:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ad6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.205463:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201562.205484:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-20, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201562.205489:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-20, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201562.205511:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-20,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201562.205674:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c06 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.205747:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c06 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.205756:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c06 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.205774:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187491200:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.205900:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.211473:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187491200:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.211498:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c06 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0af2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.211520:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c06 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x87:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0af2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.211528:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c06 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x87:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0af2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.211538:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c06 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x87:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0af2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.211672:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c06 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x87:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0af2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.211751:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c06 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x87:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0af2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.211761:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c06 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x87:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0af2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.211768:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c06 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x87:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0af2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.211793:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793d500 x1875062187491200/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.211806:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c06 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x87:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0af2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201562.211821:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c06 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x87:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0af2 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.211830:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/135 (0 0) 00000080:00200000:0.0:1788201562.211837:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201562.211879:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x87:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.211890:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.211917:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x87:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.212381:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x87:0x0](ffff8e8289e30908): rc = 0 00000080:00200000:0.0:1788201562.212397:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x87:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201562.212403:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x87:0x0](ffff8e8289e30908) for lock 0x9ae3109cee9e1c06 00000080:00010000:0.0:1788201562.212413:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x87:0x0](ffff8e8289e30908) 00000080:00010000:0.0:1788201562.212417:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1c06 from it ffffaa1802393e00 00010000:00010000:0.0:1788201562.212421:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c06 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x87:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc0af2 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.212430:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c06 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x87:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc0af2 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.212463:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.212480:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.212490:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ad6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.212500:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bff lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ad6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.212513:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201562.212517:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.212520:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1bff 00000080:00010000:0.0:1788201562.212524:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1bff from it ffffaa1802393d70 00010000:00010000:0.0:1788201562.212527:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bff lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ad6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.212535:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ad6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201562.212666:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201562.212674:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-20, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201562.212760:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bff lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0ad6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.212794:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c06 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x87:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc0af2 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.212804:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x87:0x0]: losing layout lock 00010000:00010000:0.0:1788201562.212832:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c06 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x87:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0af2 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.212840:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bff lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0ad6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.212846:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201562.212849:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c06 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x87:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0af2 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.212864:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1bff lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0ad6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201562.212881:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201562.212890:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793d500 x1875062187491456/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201562.212912:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187491456:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201562.213124:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.220356:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187491456:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201562.220417:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x87:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.220429:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.220456:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x87:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.220464:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x87:0x0](ffff8e8289e30908) 00000080:00200000:0.0:1788201562.220674:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.220754:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.220815:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c0d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.220826:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c0d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.220836:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c0d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.220855:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187491584:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.221167:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.225481:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187491584:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.225508:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c0d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b1c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.225520:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c0d lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b1c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.225531:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c0d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0b1c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.225659:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c0d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b1c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.225676:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c0d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0b1c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.225750:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c0d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0b1c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.225758:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c0d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0b1c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.225771:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793c380 x1875062187491584/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.225796:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c0d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b1c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.225815:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201562.225823:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201562.225852:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201562.225860:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1c0d 00000080:00010000:0.0:1788201562.225865:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.225869:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1c0d from it ffffaa1802393c10 00010000:00010000:0.0:1788201562.225872:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c0d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0b1c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.225881:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c0d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b1c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.225905:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201562.226217:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-21, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201562.226224:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-21, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201562.226248:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-21,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201562.226442:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c14 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.226452:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c14 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.226461:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c14 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.226485:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187491712:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.226960:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.231767:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187491712:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.231802:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c14 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b38 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.231823:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c14 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x88:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b38 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.231831:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c14 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b38 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.231842:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c14 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0b38 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.231852:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c14 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b38 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.231864:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c14 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0b38 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.231872:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c14 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0b38 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.231879:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c14 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0b38 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.231890:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793c380 x1875062187491712/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.231901:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c14 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b38 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201562.231915:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c14 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b38 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.231925:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/136 (0 0) 00000080:00200000:0.0:1788201562.232291:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201562.232416:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x88:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.232428:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.232455:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x88:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.232512:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x88:0x0](ffff8e8289e2d348): rc = 0 00000080:00200000:0.0:1788201562.232528:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x88:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201562.232534:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x88:0x0](ffff8e8289e2d348) for lock 0x9ae3109cee9e1c14 00000080:00010000:0.0:1788201562.232667:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x88:0x0](ffff8e8289e2d348) 00000080:00010000:0.0:1788201562.232672:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1c14 from it ffffaa1802393e00 00010000:00010000:0.0:1788201562.232677:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c14 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc0b38 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.232752:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c14 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc0b38 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.232797:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.232816:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.232826:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c0d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b1c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.232838:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c0d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b1c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.232850:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201562.232854:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.232857:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1c0d 00000080:00010000:0.0:1788201562.232862:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1c0d from it ffffaa1802393d70 00010000:00010000:0.0:1788201562.232865:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c0d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0b1c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.232872:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c0d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b1c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201562.232881:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201562.232888:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-21, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201562.232915:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c0d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0b1c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.233067:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c14 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc0b38 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.233079:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x88:0x0]: losing layout lock 00010000:00010000:0.0:1788201562.233113:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c14 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0b38 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.233120:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c0d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0b1c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.233126:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201562.233129:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c14 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x88:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0b38 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.233144:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c0d lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0b1c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201562.233233:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201562.233243:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793c380 x1875062187491968/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201562.233265:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187491968:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201562.233438:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.237562:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187491968:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201562.237628:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x88:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.237642:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.237669:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x88:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.237676:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x88:0x0](ffff8e8289e2d348) 00000080:00200000:0.0:1788201562.237772:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.237803:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.237856:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c1b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.237867:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c1b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.237878:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c1b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.237897:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187492096:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.238212:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.241513:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187492096:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.241539:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c1b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b62 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.241563:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c1b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b62 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.241574:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c1b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0b62 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.241585:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c1b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b62 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.241597:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c1b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0b62 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.241612:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c1b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0b62 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.241619:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c1b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0b62 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.241631:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793ca80 x1875062187492096/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.241644:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c1b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b62 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.241663:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201562.241670:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201562.241705:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201562.241711:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1c1b 00000080:00010000:0.0:1788201562.241716:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.241719:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1c1b from it ffffaa1802393c10 00010000:00010000:0.0:1788201562.241723:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c1b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0b62 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.241731:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c1b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b62 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.241754:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201562.241794:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-22, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201562.241798:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-22, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201562.241818:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-22,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201562.241855:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c22 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.241862:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c22 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.241871:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c22 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.241891:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187492224:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.241968:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.247403:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187492224:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.247432:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c22 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b7e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.247456:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c22 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x89:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b7e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.247464:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c22 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x89:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b7e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.247475:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c22 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x89:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0b7e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.247486:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c22 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x89:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b7e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.247498:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c22 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x89:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0b7e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.247507:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c22 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x89:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0b7e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.247514:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c22 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x89:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0b7e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.247528:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793ca80 x1875062187492224/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.247549:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c22 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x89:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b7e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201562.247564:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c22 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x89:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b7e expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.247575:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/137 (0 0) 00000080:00200000:0.0:1788201562.247583:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201562.247639:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x89:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.247651:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.247788:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x89:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.247853:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x89:0x0](ffff8e8289e2db88): rc = 0 00000080:00200000:0.0:1788201562.247868:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x89:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201562.247874:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x89:0x0](ffff8e8289e2db88) for lock 0x9ae3109cee9e1c22 00000080:00010000:0.0:1788201562.247883:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x89:0x0](ffff8e8289e2db88) 00000080:00010000:0.0:1788201562.247886:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1c22 from it ffffaa1802393e00 00010000:00010000:0.0:1788201562.247889:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c22 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x89:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc0b7e expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.247898:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c22 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x89:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc0b7e expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.247938:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.247956:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.247967:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c1b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b62 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.247978:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c1b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b62 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.247990:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201562.248097:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.248102:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1c1b 00000080:00010000:0.0:1788201562.248107:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1c1b from it ffffaa1802393d70 00010000:00010000:0.0:1788201562.248111:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c1b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0b62 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.248119:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c1b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0b62 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201562.248129:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201562.248137:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-22, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201562.248230:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c1b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0b62 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.248252:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c22 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x89:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc0b7e expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.248261:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x89:0x0]: losing layout lock 00010000:00010000:0.0:1788201562.248292:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c22 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x89:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0b7e expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.248300:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c1b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0b62 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.248358:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201562.248362:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1a00/0x9ae3109cee9e1c22 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x89:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0b7e expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.248379:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c1b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0b62 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201562.248397:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201562.248405:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793ca80 x1875062187492480/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201562.248428:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187492480:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201562.248493:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.253739:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187492480:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201562.253807:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x89:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.253819:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.253846:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x89:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.253853:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x89:0x0](ffff8e8289e2db88) 00000080:00200000:0.0:1788201562.253945:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.253966:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.254021:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c29 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.254030:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c29 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.254040:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c29 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.254060:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187492608:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.254133:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.261062:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187492608:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.261090:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c29 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ba8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.261103:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c29 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ba8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.261116:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c29 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ba8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.261127:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c29 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ba8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.261140:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c29 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ba8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.261157:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c29 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ba8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.261164:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c29 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ba8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.261176:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793f480 x1875062187492608/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.261189:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c29 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ba8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.261206:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201562.261212:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201562.261239:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201562.261245:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1c29 00000080:00010000:0.0:1788201562.261249:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.261252:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1c29 from it ffffaa1802393c10 00010000:00010000:0.0:1788201562.261255:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c29 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ba8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.261263:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c29 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ba8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.261284:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201562.261328:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-23, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201562.261334:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-23, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201562.261355:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-23,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201562.261394:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c30 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.261401:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c30 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.261409:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c30 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.261427:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187492736:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.261489:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.265073:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187492736:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.265100:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c30 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0bc4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.265121:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c30 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0bc4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.265129:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c30 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0bc4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.265139:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c30 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0bc4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.265162:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c30 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0bc4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.265176:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c30 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0bc4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.265184:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c30 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0bc4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.265193:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c30 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0bc4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.265205:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793f480 x1875062187492736/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.265216:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c30 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0bc4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201562.265230:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c30 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0bc4 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.265239:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/138 (0 0) 00000080:00200000:0.0:1788201562.265246:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201562.265287:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x8a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.265299:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.265343:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x8a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.265399:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x8a:0x0](ffff8e8289c0d348): rc = 0 00000080:00200000:0.0:1788201562.265415:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x8a:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201562.265421:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x8a:0x0](ffff8e8289c0d348) for lock 0x9ae3109cee9e1c30 00000080:00010000:0.0:1788201562.265431:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x8a:0x0](ffff8e8289c0d348) 00000080:00010000:0.0:1788201562.265435:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1c30 from it ffffaa1802393e00 00010000:00010000:0.0:1788201562.265438:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c30 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc0bc4 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.265447:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c30 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc0bc4 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.265480:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.265498:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.265508:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c29 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ba8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.265519:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c29 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ba8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.265532:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201562.265536:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.265540:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1c29 00000080:00010000:0.0:1788201562.265544:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1c29 from it ffffaa1802393d70 00010000:00010000:0.0:1788201562.265547:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c29 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ba8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.265555:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c29 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ba8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201562.265563:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201562.265569:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-23, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201562.265594:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c29 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0ba8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.265616:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c30 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc0bc4 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.265625:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x8a:0x0]: losing layout lock 00010000:00010000:0.0:1788201562.265651:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c30 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0bc4 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.265659:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c29 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0ba8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.265665:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201562.265668:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c30 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x8a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0bc4 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.265694:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c29 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0ba8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201562.265713:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201562.265720:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793f480 x1875062187492992/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201562.265743:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187492992:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201562.265820:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.273447:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187492992:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201562.273504:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x8a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.273515:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.273540:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x8a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.273548:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x8a:0x0](ffff8e8289c0d348) 00000080:00200000:0.0:1788201562.273644:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.273664:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.273727:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c37 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.273736:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c37 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.273745:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c37 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.273761:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187493120:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.273837:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.276592:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187493120:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.276631:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c37 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0bee expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.276643:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c37 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0bee expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.276654:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c37 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0bee expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.276665:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c37 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0bee expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.276686:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c37 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0bee expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.276695:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c37 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0bee expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.276702:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c37 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0bee expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.276714:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793e300 x1875062187493120/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.276727:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c37 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0bee expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.276746:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201562.276752:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201562.276787:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201562.276794:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1c37 00000080:00010000:0.0:1788201562.276800:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.276803:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1c37 from it ffffaa1802393c10 00010000:00010000:0.0:1788201562.276806:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c37 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0bee expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.276815:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c37 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0bee expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.276838:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201562.276860:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-24, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201562.276864:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-24, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201562.276886:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-24,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201562.276923:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c3e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.276931:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c3e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.276940:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c3e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.276968:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187493248:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.277429:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.284073:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187493248:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.284099:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c3e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0c0a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.284120:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c3e lrc: 4/1,0 mode: --/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0c0a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.284128:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c3e lrc: 4/1,0 mode: --/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0c0a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.284139:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c3e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0c0a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.284252:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c3e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0c0a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.284267:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c3e lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0c0a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.284276:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c3e lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0c0a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.284283:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c3e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0c0a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.284295:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793e300 x1875062187493248/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.284344:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c3e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0c0a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201562.284359:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c3e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0c0a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.284369:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/139 (0 0) 00000080:00200000:0.0:1788201562.284376:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201562.284419:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x8b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.284431:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.284458:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x8b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.284518:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x8b:0x0](ffff8e8289c10908): rc = 0 00000080:00200000:0.0:1788201562.284534:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x8b:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201562.284540:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x8b:0x0](ffff8e8289c10908) for lock 0x9ae3109cee9e1c3e 00000080:00010000:0.0:1788201562.284549:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x8b:0x0](ffff8e8289c10908) 00000080:00010000:0.0:1788201562.284553:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1c3e from it ffffaa1802393e00 00010000:00010000:0.0:1788201562.284556:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c3e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc0c0a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.284565:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c3e lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc0c0a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.284594:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201562.285708:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201562.285725:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c37 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0bee expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.285738:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c37 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0bee expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.285746:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201562.285754:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201562.285757:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1c37 00000080:00010000:1.0:1788201562.285762:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1c37 from it ffffaa1802393d70 00010000:00010000:1.0:1788201562.285765:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c37 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0bee expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.285773:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c37 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0bee expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201562.285796:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201562.285803:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-24, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201562.285832:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c37 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0bee expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201562.285854:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c3e lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc0c0a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.285883:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x8b:0x0]: losing layout lock 00010000:00010000:1.0:1788201562.285919:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c3e lrc: 1/0,0 mode: --/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0c0a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.285927:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c37 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0bee expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201562.285932:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201562.285935:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e1c3e lrc: 0/0,0 mode: --/PR res: [0x200006991:0x8b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0c0a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.285951:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1200/0x9ae3109cee9e1c37 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0bee expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201562.285969:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201562.285979:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289513800 x1875062187493504/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201562.286241:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187493504:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201562.286375:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1d80 going to sleep for 11 seconds 00010000:00010000:0.0:1788201562.308722:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201562.426879:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561600/0x9ae3109cee9e1c99 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0dd8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.426904:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560c00/0x9ae3109cee9e1ca0 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x92:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc0df4 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.426918:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x92:0x0]: losing layout lock 00010000:00010000:0.0:1788201562.426949:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560c00/0x9ae3109cee9e1ca0 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0df4 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.426957:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561600/0x9ae3109cee9e1c99 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0dd8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.426962:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201562.426965:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560c00/0x9ae3109cee9e1ca0 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x92:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0df4 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.426981:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561600/0x9ae3109cee9e1c99 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0dd8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201562.427012:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201562.427022:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793e300 x1875062187497088/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201562.427044:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187497088:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201562.427114:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.431834:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187497088:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201562.431897:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x92:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201562.431910:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201562.431942:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x92:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201562.431951:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x92:0x0](ffff8e8289e321c8) 00000080:00200000:1.0:1788201562.432209:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201562.432244:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201562.432300:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1ca7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.432474:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1ca7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.432487:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1ca7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201562.432509:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187497216:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201562.432633:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.435826:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187497216:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201562.435853:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1ca7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e1e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.435865:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1ca7 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e1e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.435877:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1ca7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0e1e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.435887:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1ca7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e1e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.435900:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1ca7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0e1e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.435907:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1ca7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0e1e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.435929:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1ca7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0e1e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201562.435941:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289511500 x1875062187497216/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201562.435954:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1ca7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e1e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.435967:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201562.435974:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201562.436127:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201562.436135:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1ca7 00000080:00010000:1.0:1788201562.436142:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201562.436145:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ca7 from it ffffaa1802393c10 00010000:00010000:1.0:1788201562.436149:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1ca7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0e1e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.436159:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1ca7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e1e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.436183:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201562.436216:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-32, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201562.436220:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-32, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201562.436243:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-32,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201562.436289:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cae lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.436303:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cae lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.436449:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cae lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201562.436470:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187497344:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201562.436569:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.439853:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187497344:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201562.439881:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cae lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e3a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.439904:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cae lrc: 4/1,0 mode: --/PR res: [0x200006991:0x93:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e3a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.439912:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cae lrc: 4/1,0 mode: --/PR res: [0x200006991:0x93:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e3a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.439923:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cae lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x93:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0e3a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.439934:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cae lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x93:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e3a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.439947:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cae lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x93:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0e3a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.439973:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cae lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x93:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0e3a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.439981:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cae lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x93:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0e3a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201562.439993:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289511500 x1875062187497344/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201562.440140:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cae lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x93:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e3a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201562.440153:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cae lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x93:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e3a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.440162:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/147 (0 0) 00000080:00200000:1.0:1788201562.440169:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201562.440208:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x93:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201562.440219:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201562.440247:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x93:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201562.440443:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x93:0x0](ffff8e8289e33a88): rc = 0 00000080:00200000:1.0:1788201562.440461:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x93:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:1.0:1788201562.440466:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x93:0x0](ffff8e8289e33a88) for lock 0x9ae3109cee9e1cae 00000080:00010000:1.0:1788201562.440476:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x93:0x0](ffff8e8289e33a88) 00000080:00010000:1.0:1788201562.440479:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1cae from it ffffaa1802393e00 00010000:00010000:1.0:1788201562.440483:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cae lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x93:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc0e3a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.440504:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cae lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x93:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc0e3a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.440533:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201562.440548:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201562.440557:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1ca7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e1e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.440566:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1ca7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e1e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.440573:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201562.440576:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201562.440579:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1ca7 00000080:00010000:1.0:1788201562.440582:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ca7 from it ffffaa1802393d70 00010000:00010000:1.0:1788201562.440584:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1ca7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0e1e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.440590:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1ca7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e1e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201562.440597:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201562.440664:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-32, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201562.440750:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1ca7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0e1e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201562.440773:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cae lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x93:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc0e3a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.440801:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x93:0x0]: losing layout lock 00010000:00010000:1.0:1788201562.440830:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cae lrc: 1/0,0 mode: --/PR res: [0x200006991:0x93:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0e3a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.440837:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1ca7 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0e1e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201562.440842:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201562.440844:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cae lrc: 0/0,0 mode: --/PR res: [0x200006991:0x93:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0e3a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.440857:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1ca7 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0e1e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201562.440874:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201562.440882:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289511500 x1875062187497600/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201562.440904:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187497600:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201562.441124:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.444807:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187497600:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201562.444867:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x93:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201562.444881:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201562.444909:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x93:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201562.444917:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x93:0x0](ffff8e8289e33a88) 00000080:00200000:1.0:1788201562.445134:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201562.445167:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201562.445219:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cb5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.445228:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cb5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.445238:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cb5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201562.445257:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187497728:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201562.445439:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.448288:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289513100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187497728:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201562.448389:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cb5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e64 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.448401:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cb5 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e64 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.448412:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cb5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0e64 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.448421:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cb5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e64 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.448433:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cb5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0e64 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.448440:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cb5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0e64 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.448460:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cb5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0e64 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201562.448472:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289513100 x1875062187497728/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201562.448485:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cb5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e64 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.448497:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201562.448503:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201562.448529:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201562.448536:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1cb5 00000080:00010000:1.0:1788201562.448541:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201562.448544:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1cb5 from it ffffaa1802393c10 00010000:00010000:1.0:1788201562.448548:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cb5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0e64 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.448557:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cb5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e64 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.448580:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201562.448668:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-33, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201562.448674:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-33, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201562.448760:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-33,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201562.448811:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cbc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.448820:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cbc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.448829:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cbc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201562.448849:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187497856:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201562.448957:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.452215:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289513100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187497856:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201562.452241:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cbc lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e80 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.452261:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cbc lrc: 4/1,0 mode: --/PR res: [0x200006991:0x94:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e80 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.452269:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cbc lrc: 4/1,0 mode: --/PR res: [0x200006991:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e80 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.452281:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cbc lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0e80 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.452290:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cbc lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e80 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.452302:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cbc lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0e80 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.452338:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cbc lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0e80 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.452345:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cbc lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0e80 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201562.452356:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289513100 x1875062187497856/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201562.452367:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cbc lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e80 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201562.452381:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cbc lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e80 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.452392:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/148 (0 0) 00000080:00200000:1.0:1788201562.452399:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201562.452445:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x94:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201562.452456:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201562.452481:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x94:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201562.452938:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x94:0x0](ffff8e8289e2b248): rc = 0 00000080:00200000:1.0:1788201562.452958:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x94:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:1.0:1788201562.452964:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x94:0x0](ffff8e8289e2b248) for lock 0x9ae3109cee9e1cbc 00000080:00010000:1.0:1788201562.452975:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x94:0x0](ffff8e8289e2b248) 00000080:00010000:1.0:1788201562.452978:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1cbc from it ffffaa1802393e00 00010000:00010000:1.0:1788201562.452982:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cbc lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc0e80 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.453161:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cbc lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc0e80 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.453196:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201562.453212:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201562.453223:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cb5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e64 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.453233:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cb5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e64 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.453240:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201562.453245:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201562.453248:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1cb5 00000080:00010000:1.0:1788201562.453252:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1cb5 from it ffffaa1802393d70 00010000:00010000:1.0:1788201562.453255:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cb5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0e64 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.453262:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cb5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0e64 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201562.453270:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201562.453278:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-33, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201562.453305:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cb5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0e64 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201562.453410:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cbc lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc0e80 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.453419:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x94:0x0]: losing layout lock 00010000:00010000:1.0:1788201562.453446:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cbc lrc: 1/0,0 mode: --/PR res: [0x200006991:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0e80 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.453453:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cb5 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0e64 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201562.453459:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201562.453462:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cbc lrc: 0/0,0 mode: --/PR res: [0x200006991:0x94:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0e80 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.453477:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cb5 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0e64 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201562.453494:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201562.453504:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289513100 x1875062187498112/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201562.453527:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187498112:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201562.453696:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.458416:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289513100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187498112:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201562.458482:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x94:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201562.458496:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201562.458522:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x94:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201562.458530:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x94:0x0](ffff8e8289e2b248) 00000080:00200000:1.0:1788201562.458757:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201562.458789:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201562.458847:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cc3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.458857:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cc3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.458866:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cc3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201562.458884:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187498240:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201562.459013:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.467223:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187498240:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201562.467253:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cc3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0eaa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.467361:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cc3 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0eaa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.467376:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cc3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0eaa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.467388:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cc3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0eaa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.467401:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cc3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0eaa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.467409:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cc3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0eaa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.467417:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cc3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0eaa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201562.467430:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289512300 x1875062187498240/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201562.467444:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cc3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0eaa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.467457:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201562.467465:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201562.467496:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201562.467502:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1cc3 00000080:00010000:1.0:1788201562.467507:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201562.467510:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1cc3 from it ffffaa1802393c10 00010000:00010000:1.0:1788201562.467514:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cc3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0eaa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.467523:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cc3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0eaa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.467546:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201562.467566:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-34, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201562.467570:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-34, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201562.467590:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-34,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201562.467758:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cca lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.467767:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cca lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.467787:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cca lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201562.467809:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187498368:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201562.467993:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.472395:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187498368:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201562.472424:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cca lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ec6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.472446:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cca lrc: 4/1,0 mode: --/PR res: [0x200006991:0x95:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ec6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.472454:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cca lrc: 4/1,0 mode: --/PR res: [0x200006991:0x95:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ec6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.472465:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cca lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x95:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ec6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.472475:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cca lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x95:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ec6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.472487:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cca lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x95:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ec6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.472500:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cca lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x95:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ec6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.472507:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cca lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x95:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ec6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201562.472517:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289512300 x1875062187498368/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201562.472528:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cca lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x95:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ec6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201562.472540:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cca lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x95:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ec6 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.472550:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/149 (0 0) 00000080:00200000:1.0:1788201562.472557:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201562.472665:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x95:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201562.472745:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201562.472772:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x95:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201562.472844:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x95:0x0](ffff8e8289e342c8): rc = 0 00000080:00200000:1.0:1788201562.472858:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x95:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:1.0:1788201562.472864:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x95:0x0](ffff8e8289e342c8) for lock 0x9ae3109cee9e1cca 00000080:00010000:1.0:1788201562.472875:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x95:0x0](ffff8e8289e342c8) 00000080:00010000:1.0:1788201562.472878:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1cca from it ffffaa1802393e00 00010000:00010000:1.0:1788201562.472882:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cca lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x95:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc0ec6 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.472895:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cca lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x95:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc0ec6 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.473242:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201562.473263:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201562.473375:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cc3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0eaa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.473387:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cc3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0eaa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.473393:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201562.473399:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201562.473402:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1cc3 00000080:00010000:1.0:1788201562.473406:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1cc3 from it ffffaa1802393d70 00010000:00010000:1.0:1788201562.473409:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cc3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0eaa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.473416:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cc3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0eaa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201562.473425:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201562.473433:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-34, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201562.473459:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cc3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0eaa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201562.473482:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cca lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x95:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc0ec6 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.473492:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x95:0x0]: losing layout lock 00010000:00010000:1.0:1788201562.473527:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cca lrc: 1/0,0 mode: --/PR res: [0x200006991:0x95:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0ec6 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.473534:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cc3 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0eaa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201562.473540:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201562.473543:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cca lrc: 0/0,0 mode: --/PR res: [0x200006991:0x95:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0ec6 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.473560:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cc3 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0eaa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201562.473578:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201562.473588:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289512300 x1875062187498624/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201562.473752:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187498624:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201562.473843:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.478344:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187498624:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201562.478407:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x95:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201562.478420:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201562.478447:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x95:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201562.478455:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x95:0x0](ffff8e8289e342c8) 00000080:00200000:1.0:1788201562.478541:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201562.478561:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201562.478621:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cd1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.478630:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cd1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.478639:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cd1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201562.478660:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512d80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187498752:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201562.478731:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.482827:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512d80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187498752:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201562.482855:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cd1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ef0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.482866:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cd1 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ef0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.482877:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cd1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ef0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.482886:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cd1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ef0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.482897:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cd1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ef0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.482904:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cd1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ef0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.482911:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cd1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ef0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201562.482992:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289512d80 x1875062187498752/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201562.483126:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cd1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ef0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.483140:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201562.483147:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201562.483173:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201562.483179:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1cd1 00000080:00010000:1.0:1788201562.483184:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201562.483187:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1cd1 from it ffffaa1802393c10 00010000:00010000:1.0:1788201562.483190:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cd1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ef0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.483198:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cd1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ef0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.483220:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201562.483244:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-35, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201562.483248:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-35, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201562.483368:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-35,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201562.483411:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cd8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.483419:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cd8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.483426:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cd8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201562.483445:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512d80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187498880:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201562.483561:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.487768:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512d80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187498880:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201562.487806:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cd8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f0c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.487830:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cd8 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x96:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f0c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.487838:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cd8 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f0c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.487849:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cd8 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f0c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.487861:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cd8 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f0c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.487875:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cd8 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f0c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.487889:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cd8 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f0c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.487895:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cd8 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f0c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201562.487907:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289512d80 x1875062187498880/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201562.487918:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cd8 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f0c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201562.488123:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cd8 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f0c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.488136:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/150 (0 0) 00000080:00200000:1.0:1788201562.488144:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201562.488183:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x96:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201562.488196:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201562.488226:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x96:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201562.488387:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x96:0x0](ffff8e8289e35348): rc = 0 00000080:00200000:1.0:1788201562.488402:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x96:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:1.0:1788201562.488408:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x96:0x0](ffff8e8289e35348) for lock 0x9ae3109cee9e1cd8 00000080:00010000:1.0:1788201562.488420:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x96:0x0](ffff8e8289e35348) 00000080:00010000:1.0:1788201562.488423:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1cd8 from it ffffaa1802393e00 00010000:00010000:1.0:1788201562.488427:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cd8 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc0f0c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.488439:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cd8 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc0f0c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.488469:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201562.488485:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201562.488495:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cd1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ef0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.488508:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cd1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ef0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.488514:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201562.488519:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201562.488522:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1cd1 00000080:00010000:1.0:1788201562.488525:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1cd1 from it ffffaa1802393d70 00010000:00010000:1.0:1788201562.488528:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cd1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0ef0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.488535:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cd1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0ef0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201562.488542:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201562.488550:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-35, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201562.488572:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cd1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0ef0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201562.488594:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cd8 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc0f0c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.488665:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x96:0x0]: losing layout lock 00010000:00010000:1.0:1788201562.488766:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cd8 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0f0c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.488786:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cd1 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0ef0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201562.488792:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201562.488795:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1cd8 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x96:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0f0c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.488811:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1cd1 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0ef0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201562.488828:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201562.488836:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289512d80 x1875062187499136/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201562.488859:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512d80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187499136:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201562.489185:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.499929:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512d80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187499136:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201562.499991:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x96:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201562.500015:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201562.500045:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x96:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201562.500054:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x96:0x0](ffff8e8289e35348) 00000080:00200000:1.0:1788201562.500145:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201562.500168:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201562.500224:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cdf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.500234:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cdf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.500243:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cdf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201562.500263:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187499264:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201562.500362:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.506155:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187499264:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201562.506186:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cdf lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f36 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.506199:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cdf lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f36 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.506211:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cdf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f36 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.506222:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cdf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f36 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.506234:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cdf lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f36 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.506241:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cdf lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f36 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.506249:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cdf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f36 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201562.506260:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289510700 x1875062187499264/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201562.506457:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cdf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f36 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.506470:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201562.506477:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201562.506502:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201562.506507:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1cdf 00000080:00010000:1.0:1788201562.506512:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201562.506515:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1cdf from it ffffaa1802393c10 00010000:00010000:1.0:1788201562.506518:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cdf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f36 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.506526:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cdf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f36 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.506549:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201562.506566:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-36, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201562.506570:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-36, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201562.506591:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-36,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201562.506639:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1ce6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.506801:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1ce6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.506813:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1ce6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201562.506834:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187499392:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201562.507114:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.510513:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187499392:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201562.510540:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1ce6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f52 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.510560:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1ce6 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x97:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f52 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.510569:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1ce6 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f52 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.510579:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1ce6 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f52 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.510590:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1ce6 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f52 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.510602:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1ce6 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f52 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.510616:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1ce6 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f52 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.510624:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1ce6 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f52 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201562.510635:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289510700 x1875062187499392/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201562.510647:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1ce6 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f52 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201562.510658:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1ce6 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f52 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.510668:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/151 (0 0) 00000080:00200000:1.0:1788201562.510791:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201562.510835:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x97:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201562.510846:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201562.510873:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x97:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201562.511176:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x97:0x0](ffff8e8289e29148): rc = 0 00000080:00200000:1.0:1788201562.511195:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x97:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:1.0:1788201562.511202:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x97:0x0](ffff8e8289e29148) for lock 0x9ae3109cee9e1ce6 00000080:00010000:1.0:1788201562.511212:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x97:0x0](ffff8e8289e29148) 00000080:00010000:1.0:1788201562.511215:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ce6 from it ffffaa1802393e00 00010000:00010000:1.0:1788201562.511220:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1ce6 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc0f52 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.511235:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1ce6 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc0f52 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.511498:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201562.511518:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201562.511530:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cdf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f36 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.511540:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cdf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f36 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.511547:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201562.511552:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201562.511555:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1cdf 00000080:00010000:1.0:1788201562.511559:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1cdf from it ffffaa1802393d70 00010000:00010000:1.0:1788201562.511562:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cdf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f36 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.511570:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cdf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f36 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201562.511579:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201562.511586:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-36, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201562.511621:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cdf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0f36 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201562.511641:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1ce6 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc0f52 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.511650:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x97:0x0]: losing layout lock 00010000:00010000:1.0:1788201562.511749:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1ce6 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0f52 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.511757:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cdf lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0f36 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201562.511762:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201562.511765:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1ce6 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x97:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0f52 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.511807:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cdf lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0f36 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201562.511825:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201562.511835:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289510700 x1875062187499648/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201562.511858:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187499648:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201562.512005:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.516294:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187499648:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201562.516380:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x97:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201562.516392:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201562.516421:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x97:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201562.516428:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x97:0x0](ffff8e8289e29148) 00000080:00200000:1.0:1788201562.516516:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201562.516536:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201562.516585:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1ced lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.516595:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1ced lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.516617:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1ced lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201562.516638:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187499776:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201562.516709:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.520101:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187499776:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201562.520130:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1ced lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f7c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.520142:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1ced lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f7c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.520154:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1ced lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f7c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.520166:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1ced lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f7c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.520179:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1ced lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f7c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.520187:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1ced lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f7c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.520195:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1ced lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f7c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201562.520206:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289510a80 x1875062187499776/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201562.520218:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1ced lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f7c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.520229:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201562.520235:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201562.520260:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201562.520279:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1ced 00000080:00010000:1.0:1788201562.520284:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201562.520287:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ced from it ffffaa1802393c10 00010000:00010000:1.0:1788201562.520291:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1ced lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f7c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.520300:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1ced lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f7c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.520357:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201562.520379:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-37, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201562.520384:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-37, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201562.520410:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-37,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201562.520452:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cf4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.520459:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cf4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.520468:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cf4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201562.520488:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187499904:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201562.520551:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.524104:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187499904:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201562.524131:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cf4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f98 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.524152:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cf4 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x98:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f98 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.524160:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cf4 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x98:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f98 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.524171:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cf4 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x98:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f98 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.524180:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cf4 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x98:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f98 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.524192:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cf4 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x98:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f98 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.524204:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cf4 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x98:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f98 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.524211:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cf4 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x98:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f98 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201562.524223:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289510a80 x1875062187499904/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201562.524233:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cf4 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x98:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f98 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201562.524244:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cf4 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x98:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f98 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.524253:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/152 (0 0) 00000080:00200000:1.0:1788201562.524260:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201562.524365:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x98:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201562.524376:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201562.524401:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x98:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201562.524457:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x98:0x0](ffff8e8289e2ec08): rc = 0 00000080:00200000:1.0:1788201562.524473:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x98:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:1.0:1788201562.524478:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x98:0x0](ffff8e8289e2ec08) for lock 0x9ae3109cee9e1cf4 00000080:00010000:1.0:1788201562.524488:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x98:0x0](ffff8e8289e2ec08) 00000080:00010000:1.0:1788201562.524491:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1cf4 from it ffffaa1802393e00 00010000:00010000:1.0:1788201562.524495:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cf4 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x98:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc0f98 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.524508:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cf4 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x98:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc0f98 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.524539:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201562.524557:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201562.524566:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1ced lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f7c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.524575:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1ced lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f7c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.524580:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201562.524584:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201562.524586:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1ced 00000080:00010000:1.0:1788201562.524590:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ced from it ffffaa1802393d70 00010000:00010000:1.0:1788201562.524593:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1ced lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0f7c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.524612:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1ced lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0f7c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201562.524621:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201562.524628:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-37, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201562.524652:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1ced lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0f7c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201562.524674:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cf4 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x98:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc0f98 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.524695:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x98:0x0]: losing layout lock 00010000:00010000:1.0:1788201562.524727:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cf4 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x98:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0f98 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.524733:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1ced lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0f7c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201562.524738:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201562.524741:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e1cf4 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x98:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0f98 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.524754:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1ced lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0f7c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201562.524771:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201562.524788:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289510a80 x1875062187500160/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201562.524808:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187500160:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201562.524871:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.529373:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187500160:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201562.529427:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x98:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201562.529440:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201562.529467:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x98:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201562.529474:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x98:0x0](ffff8e8289e2ec08) 00000080:00200000:1.0:1788201562.529570:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201562.529589:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201562.529966:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cfb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.529977:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cfb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.529985:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cfb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201562.530012:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187500288:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201562.530077:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.533263:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289513b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187500288:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201562.533289:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cfb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0fc2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.533302:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cfb lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0fc2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.533340:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cfb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0fc2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.533350:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cfb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0fc2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.533362:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cfb lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0fc2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.533370:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cfb lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0fc2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.533378:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cfb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0fc2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201562.533389:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289513b80 x1875062187500288/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201562.533403:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cfb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0fc2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.533415:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201562.533422:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201562.533450:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201562.533456:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1cfb 00000080:00010000:1.0:1788201562.533461:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201562.533464:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1cfb from it ffffaa1802393c10 00010000:00010000:1.0:1788201562.533467:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cfb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0fc2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.533476:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cfb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0fc2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.533499:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201562.533522:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-38, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201562.533526:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-38, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201562.533547:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-38,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201562.533584:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d02 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.533591:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d02 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.533610:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d02 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201562.533630:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187500416:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201562.533902:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.537821:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289513b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187500416:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201562.537848:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d02 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0fde expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.537867:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d02 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x99:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0fde expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.537874:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d02 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x99:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0fde expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.537884:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d02 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x99:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0fde expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.537895:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d02 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x99:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0fde expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.537907:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d02 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x99:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0fde expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.537919:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d02 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x99:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0fde expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.537926:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d02 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x99:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0fde expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201562.537936:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289513b80 x1875062187500416/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201562.537946:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d02 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x99:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0fde expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201562.537957:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d02 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x99:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0fde expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.537966:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/153 (0 0) 00000080:00200000:1.0:1788201562.537972:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201562.538009:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x99:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201562.538020:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201562.538047:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x99:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201562.538101:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x99:0x0](ffff8e8289e28908): rc = 0 00000080:00200000:1.0:1788201562.538115:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x99:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:1.0:1788201562.538120:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x99:0x0](ffff8e8289e28908) for lock 0x9ae3109cee9e1d02 00000080:00010000:1.0:1788201562.538130:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x99:0x0](ffff8e8289e28908) 00000080:00010000:1.0:1788201562.538133:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1d02 from it ffffaa1802393e00 00010000:00010000:1.0:1788201562.538137:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d02 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x99:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc0fde expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.538151:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d02 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x99:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc0fde expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.538178:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201562.538192:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201562.538200:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cfb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0fc2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.538210:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cfb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0fc2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.538215:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201562.538219:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201562.538222:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1cfb 00000080:00010000:1.0:1788201562.538224:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1cfb from it ffffaa1802393d70 00010000:00010000:1.0:1788201562.538227:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cfb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc0fc2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.538234:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cfb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc0fc2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201562.538240:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201562.538246:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-38, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201562.538268:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cfb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc0fc2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201562.538289:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d02 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x99:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc0fde expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.538297:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x99:0x0]: losing layout lock 00010000:00010000:1.0:1788201562.538373:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d02 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x99:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0fde expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.538380:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cfb lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0fc2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201562.538386:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201562.538389:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d02 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x99:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc0fde expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.538402:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1cfb lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc0fc2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201562.538417:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201562.538424:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289513b80 x1875062187500672/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201562.538444:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187500672:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201562.538500:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1300 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.542593:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289513b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187500672:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201562.542659:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x99:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201562.542673:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201562.542709:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x99:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201562.542717:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x99:0x0](ffff8e8289e28908) 00000080:00200000:1.0:1788201562.542812:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201562.542831:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201562.542884:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1d09 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.542893:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1d09 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.542901:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1d09 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201562.542921:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187500800:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201562.542989:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.546253:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187500800:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201562.546280:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1d09 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1008 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.546291:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1d09 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1008 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.546302:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1d09 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1008 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.546361:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1d09 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1008 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.546374:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1d09 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1008 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.546381:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1d09 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1008 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.546389:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1d09 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1008 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201562.546400:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289511f80 x1875062187500800/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201562.546414:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1008 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.546426:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201562.546433:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201562.546457:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201562.546463:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1d09 00000080:00010000:1.0:1788201562.546468:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201562.546471:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1d09 from it ffffaa1802393c10 00010000:00010000:1.0:1788201562.546475:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1008 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.546484:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1008 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.546508:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201562.546528:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-39, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201562.546531:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-39, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201562.546549:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-39,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201562.546588:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1d10 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.546595:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1d10 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.546617:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1d10 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201562.546636:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187500928:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201562.546708:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.555838:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187500928:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201562.555866:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1d10 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1024 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.555899:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1d10 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1024 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.555908:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1d10 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1024 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.555919:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1d10 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1024 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.555929:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1d10 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1024 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.555941:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1d10 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1024 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.555953:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1d10 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1024 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.555960:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1d10 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1024 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201562.555972:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289511f80 x1875062187500928/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201562.555983:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1d10 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1024 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201562.556008:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1d10 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1024 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.556019:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/154 (0 0) 00000080:00200000:1.0:1788201562.556026:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201562.556068:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x9a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201562.556080:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201562.556111:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x9a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201562.556169:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x9a:0x0](ffff8e8289c14b08): rc = 0 00000080:00200000:1.0:1788201562.556184:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x9a:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:1.0:1788201562.556190:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x9a:0x0](ffff8e8289c14b08) for lock 0x9ae3109cee9e1d10 00000080:00010000:1.0:1788201562.556201:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x9a:0x0](ffff8e8289c14b08) 00000080:00010000:1.0:1788201562.556205:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1d10 from it ffffaa1802393e00 00010000:00010000:1.0:1788201562.556208:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1d10 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc1024 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.556221:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1d10 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc1024 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.556253:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201562.556271:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201562.556280:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1008 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.556291:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1008 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.556297:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201562.556301:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201562.556304:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1d09 00000080:00010000:1.0:1788201562.556371:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1d09 from it ffffaa1802393d70 00010000:00010000:1.0:1788201562.556375:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1008 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.556382:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1008 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201562.556390:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201562.556398:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-39, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201562.556422:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc1008 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201562.556442:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1d10 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc1024 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201562.556451:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x9a:0x0]: losing layout lock 00010000:00010000:1.0:1788201562.556481:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1d10 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1024 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.556488:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1d09 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1008 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201562.556494:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201562.556496:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e1d10 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x9a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1024 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201562.556510:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e1d09 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1008 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201562.556526:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201562.556533:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289511f80 x1875062187501184/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201562.556552:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187501184:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201562.557092:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.560976:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187501184:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201562.561126:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x9a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201562.561139:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201562.561166:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x9a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201562.561173:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x9a:0x0](ffff8e8289c14b08) 00000080:00200000:1.0:1788201562.561267:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201562.561291:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201562.561444:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d17 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.561455:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d17 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.561465:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d17 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201562.561485:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512a00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187501312:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201562.561619:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.564830:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512a00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187501312:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.564857:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d17 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc104e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.565157:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d17 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc104e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.565178:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d17 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc104e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.565190:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d17 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc104e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.565203:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d17 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc104e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.565212:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d17 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc104e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.565219:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d17 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc104e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.565231:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289512a00 x1875062187501312/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.565245:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d17 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc104e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.565257:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201562.565264:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201562.565292:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201562.565298:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1d17 00000080:00010000:0.0:1788201562.565303:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.565439:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1d17 from it ffffaa1802393c10 00010000:00010000:0.0:1788201562.565445:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d17 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc104e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.565455:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d17 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc104e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.565481:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201562.565586:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-40, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201562.565591:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-40, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201562.565626:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-40,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201562.565667:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d1e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.565684:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d1e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.565693:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d1e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.565712:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187501440:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.565798:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.568653:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187501440:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.568686:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d1e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc106a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.568707:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d1e lrc: 4/1,0 mode: --/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc106a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.568720:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d1e lrc: 4/1,0 mode: --/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc106a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.568731:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d1e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc106a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.568741:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d1e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc106a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.568752:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d1e lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc106a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.568760:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d1e lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc106a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.568768:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d1e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc106a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.568790:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793fb80 x1875062187501440/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.568801:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d1e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc106a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201562.568813:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d1e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc106a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.568828:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/155 (0 0) 00000080:00200000:0.0:1788201562.568834:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201562.568873:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x9b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.568884:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.568909:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x9b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.568963:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x9b:0x0](ffff8e8289c0db88): rc = 0 00000080:00200000:0.0:1788201562.568977:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x9b:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201562.568982:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x9b:0x0](ffff8e8289c0db88) for lock 0x9ae3109cee9e1d1e 00000080:00010000:0.0:1788201562.568991:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x9b:0x0](ffff8e8289c0db88) 00000080:00010000:0.0:1788201562.569257:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1d1e from it ffffaa1802393e00 00010000:00010000:0.0:1788201562.569263:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d1e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc106a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.569273:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d1e lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc106a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.569304:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.569409:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.569420:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d17 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc104e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.569430:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d17 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc104e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.569436:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201562.569441:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.569444:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1d17 00000080:00010000:0.0:1788201562.569448:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1d17 from it ffffaa1802393d70 00010000:00010000:0.0:1788201562.569451:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d17 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc104e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.569457:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d17 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc104e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201562.569466:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201562.569472:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-40, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201562.569496:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d17 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc104e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.569517:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d1e lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc106a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.569525:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x9b:0x0]: losing layout lock 00010000:00010000:0.0:1788201562.569554:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d1e lrc: 1/0,0 mode: --/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc106a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.569561:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d17 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc104e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.569567:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201562.569569:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d1e lrc: 0/0,0 mode: --/PR res: [0x200006991:0x9b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc106a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.569584:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e1d17 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc104e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201562.569608:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201562.569616:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793fb80 x1875062187501696/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201562.569638:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187501696:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201562.569855:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.575228:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187501696:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201562.575292:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x9b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.575406:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.575437:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x9b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.575445:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x9b:0x0](ffff8e8289c0db88) 00000080:00200000:0.0:1788201562.575535:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.575555:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.575738:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1d25 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.575749:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1d25 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.575761:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1d25 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.575791:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187501824:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.575891:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.578764:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187501824:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.578799:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1d25 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1094 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.578812:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1d25 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1094 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.578823:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1d25 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1094 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.578832:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1d25 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1094 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.578843:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1d25 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1094 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.578850:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1d25 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1094 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.578856:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1d25 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1094 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.578866:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793e680 x1875062187501824/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.578877:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1d25 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1094 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.578888:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201562.578894:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201562.578915:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201562.578921:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1d25 00000080:00010000:0.0:1788201562.578926:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.578929:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1d25 from it ffffaa1802393c10 00010000:00010000:0.0:1788201562.578932:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1d25 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1094 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.578940:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1d25 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1094 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.578962:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201562.578983:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-41, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201562.578988:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-41, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201562.579141:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-41,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201562.579183:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1d2c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.579189:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1d2c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.579198:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1d2c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.579215:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187501952:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.579352:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.582665:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187501952:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.582747:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1d2c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc10b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.582785:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1d2c lrc: 4/1,0 mode: --/PR res: [0x200006991:0x9c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc10b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.582794:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1d2c lrc: 4/1,0 mode: --/PR res: [0x200006991:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc10b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.582805:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1d2c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc10b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.582816:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1d2c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc10b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.582828:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1d2c lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc10b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.582837:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1d2c lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc10b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.582845:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1d2c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc10b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.582856:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793e680 x1875062187501952/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.582866:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1d2c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc10b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201562.582885:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1d2c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc10b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.582895:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/156 (0 0) 00000080:00200000:0.0:1788201562.582902:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201562.582940:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x9c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.582951:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.582976:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x9c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.583143:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x9c:0x0](ffff8e8289e2f448): rc = 0 00000080:00200000:0.0:1788201562.583159:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x9c:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201562.583164:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x9c:0x0](ffff8e8289e2f448) for lock 0x9ae3109cee9e1d2c 00000080:00010000:0.0:1788201562.583174:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x9c:0x0](ffff8e8289e2f448) 00000080:00010000:0.0:1788201562.583177:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1d2c from it ffffaa1802393e00 00010000:00010000:0.0:1788201562.583180:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1d2c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc10b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.583189:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1d2c lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc10b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.583219:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.583236:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.583245:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1d25 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1094 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.583255:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1d25 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1094 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.583266:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201562.583271:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.583274:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1d25 00000080:00010000:0.0:1788201562.583277:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1d25 from it ffffaa1802393d70 00010000:00010000:0.0:1788201562.583281:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1d25 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1094 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.583288:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1d25 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1094 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201562.583297:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201562.583304:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-41, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201562.583405:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1d25 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc1094 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.583427:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1d2c lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc10b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.583437:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x9c:0x0]: losing layout lock 00010000:00010000:0.0:1788201562.583465:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1d2c lrc: 1/0,0 mode: --/PR res: [0x200006991:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc10b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.583474:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1d25 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1094 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.583480:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201562.583483:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e1d2c lrc: 0/0,0 mode: --/PR res: [0x200006991:0x9c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc10b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.583499:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1d25 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1094 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201562.583518:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201562.583525:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793e680 x1875062187502208/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201562.583547:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187502208:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201562.583972:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.588115:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187502208:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201562.588186:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x9c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.588201:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.588234:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x9c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.588242:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x9c:0x0](ffff8e8289e2f448) 00000080:00200000:0.0:1788201562.588358:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.588378:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.588532:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d33 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.588543:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d33 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.588553:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1c00/0x9ae3109cee9e1d33 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.588575:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187502336:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.588830:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.696829:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.733744:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d702300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187507328:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201562.733814:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xa6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201562.733827:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201562.733857:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xa6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201562.733864:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xa6:0x0](ffff8e8289e39148) 00000080:00200000:3.0:1788201562.733962:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201562.733984:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201562.734055:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1dbf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.734064:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1dbf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.734074:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1dbf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201562.734092:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952b480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187507456:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201562.734163:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.736943:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952b480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187507456:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201562.736968:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1dbf lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1396 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.736980:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1dbf lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1396 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.736993:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1dbf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1396 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.737057:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1dbf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1396 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.737069:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1dbf lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1396 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.737077:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1dbf lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1396 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.737083:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1dbf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1396 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201562.737094:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828952b480 x1875062187507456/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201562.737106:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1dbf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1396 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.737118:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201562.737123:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201562.737149:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201562.737155:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1dbf 00000080:00010000:3.0:1788201562.737160:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201562.737163:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1dbf from it ffffaa1802393c10 00010000:00010000:3.0:1788201562.737166:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1dbf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1396 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.737180:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1dbf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1396 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.737203:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201562.737235:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-52, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201562.737239:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-52, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201562.737259:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-52,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201562.737299:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1dc6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.737417:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1dc6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.737428:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1dc6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201562.737451:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952b480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187507584:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201562.737575:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.740819:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952b480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187507584:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201562.740844:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1dc6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc13b2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.740865:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1dc6 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc13b2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.740874:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1dc6 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc13b2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.740884:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1dc6 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc13b2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.740895:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1dc6 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc13b2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.740907:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1dc6 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc13b2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.740914:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1dc6 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc13b2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.740921:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1dc6 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc13b2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201562.740931:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828952b480 x1875062187507584/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201562.740942:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1dc6 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc13b2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201562.740953:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1dc6 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc13b2 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.740962:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/167 (0 0) 00000080:00200000:3.0:1788201562.740969:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201562.741110:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xa7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201562.741121:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201562.741148:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xa7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201562.741208:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xa7:0x0](ffff8e8289c0cb08): rc = 0 00000080:00200000:3.0:1788201562.741224:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xa7:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201562.741229:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xa7:0x0](ffff8e8289c0cb08) for lock 0x9ae3109cee9e1dc6 00000080:00010000:3.0:1788201562.741239:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xa7:0x0](ffff8e8289c0cb08) 00000080:00010000:3.0:1788201562.741242:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1dc6 from it ffffaa1802393e00 00010000:00010000:3.0:1788201562.741245:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1dc6 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc13b2 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.741253:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1dc6 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc13b2 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.741282:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201562.741298:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201562.741433:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1dbf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1396 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.741446:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1dbf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1396 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.741453:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201562.741458:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201562.741462:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1dbf 00000080:00010000:3.0:1788201562.741466:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1dbf from it ffffaa1802393d70 00010000:00010000:3.0:1788201562.741469:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1dbf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1396 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.741481:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1dbf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1396 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201562.741490:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201562.741497:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-52, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201562.741522:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1dbf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc1396 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201562.741545:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1dc6 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc13b2 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.741554:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xa7:0x0]: losing layout lock 00010000:00010000:3.0:1788201562.741583:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1dc6 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc13b2 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.741590:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1dbf lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1396 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201562.741596:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201562.741598:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1dc6 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xa7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc13b2 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.741743:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1dbf lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1396 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201562.741765:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201562.741783:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828952b480 x1875062187507840/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201562.741806:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952b480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187507840:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201562.741935:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.745239:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952b480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187507840:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201562.745299:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xa7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201562.745371:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201562.745396:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xa7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201562.745403:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xa7:0x0](ffff8e8289c0cb08) 00000080:00200000:3.0:1788201562.745493:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201562.745515:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201562.745566:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e1dcd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.745575:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e1dcd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.745585:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e1dcd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201562.745724:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952a300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187507968:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201562.745804:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.749167:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952a300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187507968:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201562.749195:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e1dcd lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc13dc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.749206:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e1dcd lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc13dc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.749222:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e1dcd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc13dc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.749232:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e1dcd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc13dc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.749243:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e1dcd lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc13dc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.749250:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e1dcd lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc13dc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.749257:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e1dcd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc13dc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201562.749269:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828952a300 x1875062187507968/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201562.749283:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e1dcd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc13dc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.749296:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201562.749302:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201562.749487:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201562.749496:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1dcd 00000080:00010000:0.0:1788201562.749501:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.749504:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1dcd from it ffffaa1802393c10 00010000:00010000:0.0:1788201562.749507:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e1dcd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc13dc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.749516:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e1dcd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc13dc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.749539:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201562.749559:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-53, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201562.749564:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-53, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201562.749582:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-53,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201562.749748:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1dd4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.749756:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1dd4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.749765:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1dd4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201562.749796:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187508096:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201562.749922:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000080:00200000:0.0:1788201562.758528:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-53, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201562.758579:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e1dcd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc13dc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.758762:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1dd4 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xa8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc13f8 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.758784:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xa8:0x0]: losing layout lock 00010000:00010000:0.0:1788201562.758824:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1dd4 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xa8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc13f8 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.758830:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e1dcd lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc13dc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.758835:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201562.758986:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e1dd4 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xa8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc13f8 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.759120:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e1dcd lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc13dc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201562.759144:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201562.759157:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793f100 x1875062187508352/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201562.759180:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187508352:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201562.759499:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:1.0:1788201562.923173:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b6c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.923957:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d701880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187514240:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201562.923982:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e1e7c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1740 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.924162:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e1e7c lrc: 4/1,0 mode: --/PR res: [0x200006991:0xb4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1740 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.924172:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e1e7c lrc: 4/1,0 mode: --/PR res: [0x200006991:0xb4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1740 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.924183:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e1e7c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xb4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1740 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.924194:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e1e7c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xb4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1740 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.924206:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e1e7c lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xb4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1740 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.924214:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e1e7c lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xb4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1740 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.924221:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e1e7c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xb4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1740 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201562.924233:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d701880 x1875062187514240/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201562.924244:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e1e7c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xb4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1740 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201562.924262:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e1e7c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xb4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1740 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.924273:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/180 (0 0) 00000080:00200000:0.0:1788201562.924281:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201562.928448:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xb4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201562.928463:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201562.928488:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xb4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201562.928550:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xb4:0x0](ffff8e8289e39988): rc = 0 00000080:00200000:0.0:1788201562.928567:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xb4:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201562.928575:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xb4:0x0](ffff8e8289e39988) for lock 0x9ae3109cee9e1e7c 00000080:00010000:0.0:1788201562.928587:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xb4:0x0](ffff8e8289e39988) 00000080:00010000:0.0:1788201562.928590:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1e7c from it ffffaa1802393e00 00010000:00010000:0.0:1788201562.928654:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e1e7c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xb4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc1740 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201562.928713:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e1e7c lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xb4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc1740 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.928748:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201562.928765:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201562.928788:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560a00/0x9ae3109cee9e1e75 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1724 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.928801:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560a00/0x9ae3109cee9e1e75 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1724 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.928815:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201562.928820:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201562.928824:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1e75 00000080:00010000:0.0:1788201562.928827:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1e75 from it ffffaa1802393d70 00010000:00010000:0.0:1788201562.928831:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560a00/0x9ae3109cee9e1e75 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1724 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.928839:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560a00/0x9ae3109cee9e1e75 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1724 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201562.928847:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201562.928854:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-65, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201562.928877:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560a00/0x9ae3109cee9e1e75 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc1724 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.928900:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e1e7c lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xb4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc1740 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201562.928908:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xb4:0x0]: losing layout lock 00010000:00010000:0.0:1788201562.928941:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e1e7c lrc: 1/0,0 mode: --/PR res: [0x200006991:0xb4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1740 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.928949:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560a00/0x9ae3109cee9e1e75 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1724 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201562.928954:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201562.928957:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e1e7c lrc: 0/0,0 mode: --/PR res: [0x200006991:0xb4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1740 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201562.928974:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560a00/0x9ae3109cee9e1e75 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1724 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201562.928993:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201562.929109:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793ea00 x1875062187514496/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201562.929135:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ea00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187514496:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201562.929293:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201562.932610:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ea00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187514496:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201562.932846:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xb4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201562.932921:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201562.932956:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xb4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201562.932965:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xb4:0x0](ffff8e8289e39988) 00000080:00200000:1.0:1788201562.933087:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201562.933109:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201562.933162:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e1e83 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.933173:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e1e83 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201562.933184:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e1e83 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201562.933203:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187514624:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201562.933288:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1a80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.935815:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187514624:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201562.935841:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e1e83 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc176a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.935857:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e1e83 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc176a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.935866:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e1e83 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc176a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.935875:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e1e83 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc176a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.935886:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e1e83 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc176a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.935893:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e1e83 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc176a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.935899:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e1e83 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc176a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201562.935909:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289510380 x1875062187514624/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201562.935922:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e1e83 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc176a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.935933:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201562.935939:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201562.935964:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201562.935971:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1e83 00000080:00010000:3.0:1788201562.935976:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201562.935979:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1e83 from it ffffaa1802393c10 00010000:00010000:3.0:1788201562.935983:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e1e83 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc176a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.935991:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e1e83 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc176a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.936031:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201562.936052:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-66, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201562.936056:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-66, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201562.936076:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-66,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201562.936118:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e8a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.936125:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e8a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.936133:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e8a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201562.936153:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952b800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187514752:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201562.936228:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.939498:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952b800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187514752:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201562.939523:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e8a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1786 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.939550:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e8a lrc: 4/1,0 mode: --/PR res: [0x200006991:0xb5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1786 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.939558:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e8a lrc: 4/1,0 mode: --/PR res: [0x200006991:0xb5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1786 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.939569:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e8a lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xb5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1786 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.939580:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e8a lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xb5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1786 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.939591:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e8a lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xb5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1786 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.939611:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e8a lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xb5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1786 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.939618:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e8a lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xb5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1786 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201562.939629:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828952b800 x1875062187514752/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201562.939640:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e8a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xb5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1786 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201562.939659:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e8a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xb5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1786 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.939669:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/181 (0 0) 00000080:00200000:3.0:1788201562.939750:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201562.939810:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xb5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201562.939822:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201562.939851:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xb5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201562.939914:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xb5:0x0](ffff8e8289e3cb08): rc = 0 00000080:00200000:3.0:1788201562.939930:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xb5:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201562.939935:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xb5:0x0](ffff8e8289e3cb08) for lock 0x9ae3109cee9e1e8a 00000080:00010000:3.0:1788201562.939945:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xb5:0x0](ffff8e8289e3cb08) 00000080:00010000:3.0:1788201562.939949:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1e8a from it ffffaa1802393e00 00010000:00010000:3.0:1788201562.939952:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e8a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xb5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc1786 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.939961:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e8a lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xb5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc1786 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.939990:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201562.940274:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201562.940286:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e1e83 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc176a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.940297:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e1e83 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc176a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.940396:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201562.940401:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201562.940404:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1e83 00000080:00010000:3.0:1788201562.940408:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1e83 from it ffffaa1802393d70 00010000:00010000:3.0:1788201562.940411:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e1e83 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc176a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.940419:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e1e83 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc176a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201562.940428:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201562.940434:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-66, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201562.940461:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e1e83 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc176a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201562.940484:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e8a lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xb5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc1786 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.940493:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xb5:0x0]: losing layout lock 00010000:00010000:3.0:1788201562.940525:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e8a lrc: 1/0,0 mode: --/PR res: [0x200006991:0xb5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1786 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.940532:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e1e83 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc176a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201562.940537:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201562.940540:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e8a lrc: 0/0,0 mode: --/PR res: [0x200006991:0xb5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1786 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.940557:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e1e83 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc176a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201562.940577:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201562.940586:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828952b800 x1875062187515008/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201562.940919:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952b800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187515008:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201562.941037:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.945162:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952b800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187515008:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201562.945226:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xb5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201562.945241:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201562.945273:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xb5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201562.945281:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xb5:0x0](ffff8e8289e3cb08) 00000080:00200000:3.0:1788201562.945424:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201562.945446:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201562.945506:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1e91 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.945515:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1e91 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.945525:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1e91 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201562.945544:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289528e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187515136:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201562.945635:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.948838:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289528e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187515136:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201562.948862:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1e91 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.948874:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1e91 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.948886:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1e91 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc17b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.948896:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1e91 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.948908:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1e91 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc17b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.948916:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1e91 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc17b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.948923:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1e91 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc17b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201562.948933:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289528e00 x1875062187515136/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201562.948946:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1e91 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.948963:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201562.948969:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201562.948992:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201562.949008:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1e91 00000080:00010000:3.0:1788201562.949013:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201562.949016:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1e91 from it ffffaa1802393c10 00010000:00010000:3.0:1788201562.949020:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1e91 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc17b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.949028:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1e91 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.949049:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201562.949069:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-67, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201562.949073:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-67, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201562.949090:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-67,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201562.949122:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e98 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.949129:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e98 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.949137:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e98 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201562.949153:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289528e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187515264:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201562.949223:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.952653:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289528e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187515264:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201562.952679:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e98 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17cc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.952702:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e98 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xb6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17cc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.952710:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e98 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17cc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.952722:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e98 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc17cc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.952734:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e98 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17cc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.952746:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e98 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc17cc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.952755:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e98 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc17cc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.952762:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e98 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc17cc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201562.952786:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289528e00 x1875062187515264/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201562.952798:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e98 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17cc expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201562.952812:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e98 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17cc expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.952822:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/182 (0 0) 00000080:00200000:3.0:1788201562.952828:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201562.952867:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xb6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201562.952879:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201562.952903:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xb6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201562.952955:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xb6:0x0](ffff8e8289e4ba88): rc = 0 00000080:00200000:3.0:1788201562.952967:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xb6:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201562.952972:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xb6:0x0](ffff8e8289e4ba88) for lock 0x9ae3109cee9e1e98 00000080:00010000:3.0:1788201562.952981:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xb6:0x0](ffff8e8289e4ba88) 00000080:00010000:3.0:1788201562.952985:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1e98 from it ffffaa1802393e00 00010000:00010000:3.0:1788201562.952988:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e98 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc17cc expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.953015:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e98 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc17cc expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.953044:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201562.953061:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201562.953070:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1e91 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.953081:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1e91 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.953093:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201562.953096:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201562.953099:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1e91 00000080:00010000:3.0:1788201562.953103:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1e91 from it ffffaa1802393d70 00010000:00010000:3.0:1788201562.953106:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1e91 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc17b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.953113:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1e91 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201562.953121:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201562.953127:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-67, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201562.953156:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1e91 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc17b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201562.953176:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e98 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xb6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc17cc expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.953184:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xb6:0x0]: losing layout lock 00010000:00010000:3.0:1788201562.953210:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e98 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc17cc expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.953217:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1e91 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc17b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201562.953223:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201562.953226:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1e98 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xb6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc17cc expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.953242:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1e91 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc17b0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201562.953257:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201562.953264:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289528e00 x1875062187515520/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201562.953284:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289528e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187515520:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201562.953400:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.958694:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289528e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187515520:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201562.958762:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xb6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201562.958785:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201562.958812:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xb6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201562.958821:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xb6:0x0](ffff8e8289e4ba88) 00000080:00200000:3.0:1788201562.958922:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201562.958944:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201562.959093:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1e9f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.959104:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1e9f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.959116:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1e9f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201562.959140:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187515648:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201562.959289:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.961787:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187515648:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201562.961815:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1e9f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17f6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.961827:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1e9f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17f6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.961838:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1e9f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc17f6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.961848:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1e9f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17f6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.961860:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1e9f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc17f6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.961869:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1e9f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc17f6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.961877:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1e9f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc17f6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201562.961890:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289529180 x1875062187515648/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201562.961904:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1e9f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17f6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.961923:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201562.961930:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201562.961953:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201562.961958:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1e9f 00000080:00010000:3.0:1788201562.961963:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201562.961967:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1e9f from it ffffaa1802393c10 00010000:00010000:3.0:1788201562.961970:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1e9f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc17f6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.961977:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1e9f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17f6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.962000:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201562.962022:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-68, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201562.962027:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-68, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201562.962047:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-68,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201562.962086:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ea6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.962093:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ea6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.962102:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ea6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201562.962120:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187515776:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201562.962250:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.965020:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187515776:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201562.965044:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ea6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1812 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.965065:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ea6 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xb7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1812 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.965072:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ea6 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xb7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1812 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.965081:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ea6 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xb7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1812 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.965092:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ea6 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xb7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1812 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.965104:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ea6 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xb7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1812 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.965112:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ea6 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xb7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1812 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.965120:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ea6 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xb7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1812 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201562.965131:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289529180 x1875062187515776/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201562.965142:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ea6 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xb7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1812 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201562.965155:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ea6 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xb7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1812 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.965164:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/183 (0 0) 00000080:00200000:3.0:1788201562.965171:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201562.965207:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xb7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201562.965217:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201562.965241:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xb7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201562.965294:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xb7:0x0](ffff8e8289e4c2c8): rc = 0 00000080:00200000:3.0:1788201562.965341:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xb7:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201562.965347:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xb7:0x0](ffff8e8289e4c2c8) for lock 0x9ae3109cee9e1ea6 00000080:00010000:3.0:1788201562.965356:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xb7:0x0](ffff8e8289e4c2c8) 00000080:00010000:3.0:1788201562.965360:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ea6 from it ffffaa1802393e00 00010000:00010000:3.0:1788201562.965363:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ea6 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xb7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc1812 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.965373:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ea6 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xb7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc1812 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.965399:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201562.965413:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201562.965422:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1e9f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17f6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.965432:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1e9f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17f6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.965445:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201562.965448:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201562.965452:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1e9f 00000080:00010000:3.0:1788201562.965455:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1e9f from it ffffaa1802393d70 00010000:00010000:3.0:1788201562.965458:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1e9f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc17f6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.965466:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1e9f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc17f6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201562.965474:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201562.965480:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-68, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201562.965502:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1e9f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc17f6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201562.965521:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ea6 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xb7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc1812 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.965530:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xb7:0x0]: losing layout lock 00010000:00010000:3.0:1788201562.965554:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ea6 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xb7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1812 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.965562:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1e9f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc17f6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201562.965568:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201562.965571:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ea6 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xb7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1812 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.965585:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1e9f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc17f6 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201562.965612:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201562.965619:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289529180 x1875062187516032/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201562.965638:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187516032:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201562.965708:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.968913:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187516032:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201562.968968:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xb7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201562.968980:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201562.969020:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xb7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201562.969027:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xb7:0x0](ffff8e8289e4c2c8) 00000080:00200000:3.0:1788201562.969108:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201562.969127:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201562.969178:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ead lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.969187:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ead lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.969196:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ead lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201562.969221:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952a680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187516160:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201562.969301:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.971734:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952a680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187516160:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201562.971760:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ead lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc183c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.971784:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ead lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc183c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.971795:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ead lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc183c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.971806:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ead lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc183c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.971818:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ead lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc183c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.971827:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ead lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc183c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.971834:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ead lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc183c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201562.971846:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828952a680 x1875062187516160/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201562.971860:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ead lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc183c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.971878:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201562.971884:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201562.971909:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201562.971916:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1ead 00000080:00010000:3.0:1788201562.971921:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201562.971924:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ead from it ffffaa1802393c10 00010000:00010000:3.0:1788201562.971928:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ead lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc183c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.971937:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ead lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc183c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.971961:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201562.971980:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-69, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201562.971985:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-69, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201562.972017:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-69,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201562.972059:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1eb4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.972067:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1eb4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.972075:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1eb4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201562.972093:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952a680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187516288:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201562.972174:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.975510:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952a680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187516288:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201562.975538:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1eb4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1858 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.975559:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1eb4 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xb8:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1858 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.975567:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1eb4 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1858 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.975579:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1eb4 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1858 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.975589:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1eb4 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1858 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.975612:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1eb4 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1858 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.975621:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1eb4 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1858 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.975628:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1eb4 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1858 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201562.975640:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828952a680 x1875062187516288/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201562.975652:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1eb4 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1858 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201562.975673:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1eb4 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1858 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.975683:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/184 (0 0) 00000080:00200000:3.0:1788201562.975690:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201562.975729:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xb8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201562.975740:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201562.975764:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xb8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201562.975831:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xb8:0x0](ffff8e8289e4d348): rc = 0 00000080:00200000:3.0:1788201562.975848:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xb8:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201562.975854:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xb8:0x0](ffff8e8289e4d348) for lock 0x9ae3109cee9e1eb4 00000080:00010000:3.0:1788201562.975864:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xb8:0x0](ffff8e8289e4d348) 00000080:00010000:3.0:1788201562.975867:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1eb4 from it ffffaa1802393e00 00010000:00010000:3.0:1788201562.975870:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1eb4 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc1858 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.975879:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1eb4 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc1858 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.975911:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201562.975929:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201562.975938:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ead lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc183c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.975948:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ead lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc183c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.975959:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201562.975964:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201562.975966:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1ead 00000080:00010000:3.0:1788201562.975970:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ead from it ffffaa1802393d70 00010000:00010000:3.0:1788201562.975973:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ead lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc183c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.975980:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ead lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc183c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201562.975988:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201562.976008:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-69, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201562.976035:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ead lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc183c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201562.976057:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1eb4 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xb8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc1858 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.976068:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xb8:0x0]: losing layout lock 00010000:00010000:3.0:1788201562.976097:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1eb4 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1858 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.976105:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ead lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc183c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201562.976112:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201562.976115:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1eb4 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xb8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1858 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.976132:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ead lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc183c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201562.976149:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201562.976156:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828952a680 x1875062187516544/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201562.976178:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952a680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187516544:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201562.976248:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.979578:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952a680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187516544:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201562.979645:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xb8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201562.979656:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201562.979688:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xb8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201562.979695:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xb8:0x0](ffff8e8289e4d348) 00000080:00200000:3.0:1788201562.979785:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201562.979809:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201562.979861:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e1ebb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.979870:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e1ebb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.979880:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e1ebb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201562.979898:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187516672:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201562.980026:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.981592:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187516672:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201562.981658:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e1ebb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1882 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.981675:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e1ebb lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1882 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.981685:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e1ebb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1882 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.981696:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e1ebb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1882 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.981706:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e1ebb lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1882 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.981713:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e1ebb lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1882 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.981719:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e1ebb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1882 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201562.981729:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289529880 x1875062187516672/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201562.981741:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e1ebb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1882 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.981758:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201562.981763:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201562.981795:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201562.981801:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1ebb 00000080:00010000:3.0:1788201562.981805:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201562.981808:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ebb from it ffffaa1802393c10 00010000:00010000:3.0:1788201562.981811:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e1ebb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1882 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.981819:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e1ebb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1882 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.981841:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201562.981862:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-70, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201562.981867:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-70, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201562.981886:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-70,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201562.981920:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1ec2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.981927:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1ec2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.981936:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1ec2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201562.982008:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187516800:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201562.982228:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.984171:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187516800:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201562.984191:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1ec2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc189e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.984205:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1ec2 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xb9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc189e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.984211:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1ec2 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xb9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc189e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.984218:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1ec2 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xb9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc189e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.984225:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1ec2 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xb9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc189e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.984232:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1ec2 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xb9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc189e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.984237:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1ec2 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xb9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc189e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.984242:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1ec2 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xb9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc189e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201562.984250:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289529880 x1875062187516800/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201562.984266:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1ec2 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xb9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc189e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201562.984276:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e1ec2 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xb9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc189e expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.984283:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/185 (0 0) 00000080:00200000:3.0:1788201562.984288:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201562.992423:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xba:0x0](ffff8e8289e41148): rc = 0 00000080:00200000:3.0:1788201562.992436:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xba:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201562.992440:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xba:0x0](ffff8e8289e41148) for lock 0x9ae3109cee9e1ed0 00000080:00010000:3.0:1788201562.992447:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xba:0x0](ffff8e8289e41148) 00000080:00010000:3.0:1788201562.992449:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ed0 from it ffffaa1802393e00 00010000:00010000:3.0:1788201562.992452:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560000/0x9ae3109cee9e1ed0 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xba:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc18e4 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.992458:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560000/0x9ae3109cee9e1ed0 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xba:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc18e4 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.992480:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201562.992492:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201562.992498:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560e00/0x9ae3109cee9e1ec9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc18c8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.992504:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560e00/0x9ae3109cee9e1ec9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc18c8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.992509:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201562.992512:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201562.992515:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1ec9 00000080:00010000:3.0:1788201562.992519:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ec9 from it ffffaa1802393d70 00010000:00010000:3.0:1788201562.992522:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560e00/0x9ae3109cee9e1ec9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc18c8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.992529:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560e00/0x9ae3109cee9e1ec9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc18c8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201562.992538:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201562.992544:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-71, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201562.992566:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560e00/0x9ae3109cee9e1ec9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc18c8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201562.992583:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560000/0x9ae3109cee9e1ed0 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xba:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc18e4 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201562.992588:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xba:0x0]: losing layout lock 00010000:00010000:3.0:1788201562.992620:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560000/0x9ae3109cee9e1ed0 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xba:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc18e4 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.992625:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560e00/0x9ae3109cee9e1ec9 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc18c8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201562.992628:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201562.992630:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560000/0x9ae3109cee9e1ed0 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xba:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc18e4 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201562.992641:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560e00/0x9ae3109cee9e1ec9 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc18c8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201562.992656:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201562.992661:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289528000 x1875062187517568/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201562.992781:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289528000 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187517568:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201562.992846:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201562.997458:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289528000 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187517568:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201562.997515:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xba:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201562.997528:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201562.997555:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xba:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201562.997563:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xba:0x0](ffff8e8289e41148) 00000080:00200000:3.0:1788201562.997719:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201562.997738:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201562.997801:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ed7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.997811:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ed7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201562.997821:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ed7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201562.997840:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952aa00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187517696:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201562.997916:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.004742:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952aa00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187517696:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201563.004775:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ed7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc190e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.004787:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ed7 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc190e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.004798:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ed7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc190e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.004808:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ed7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc190e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.004818:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ed7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc190e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.004827:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ed7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc190e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.004834:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ed7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc190e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201563.004844:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828952aa00 x1875062187517696/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201578 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201563.004857:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ed7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc190e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.004868:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201563.004874:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201563.004900:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201563.004907:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1ed7 00000080:00010000:3.0:1788201563.004912:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201563.004914:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ed7 from it ffffaa1802393c10 00010000:00010000:3.0:1788201563.004918:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ed7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc190e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.004925:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ed7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc190e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.004947:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201563.004967:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-72, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201563.004972:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-72, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201563.005001:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-72,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201563.005040:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1ede lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.005047:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1ede lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.005056:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1ede lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201563.005075:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952aa00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187517824:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201563.005152:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.013958:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952aa00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187517824:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201563.014004:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1ede lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc192a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.014034:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1ede lrc: 4/1,0 mode: --/PR res: [0x200006991:0xbb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc192a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.014042:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1ede lrc: 4/1,0 mode: --/PR res: [0x200006991:0xbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc192a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.014053:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1ede lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc192a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.014064:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1ede lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc192a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.014077:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1ede lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc192a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.014084:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1ede lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc192a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.014090:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1ede lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc192a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201563.014102:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828952aa00 x1875062187517824/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201563.014113:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1ede lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc192a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201563.014131:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1ede lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc192a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.014141:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/187 (0 0) 00000080:00200000:3.0:1788201563.014148:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201563.014189:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xbb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201563.014199:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201563.014227:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xbb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201563.014290:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xbb:0x0](ffff8e8289e46c08): rc = 0 00000080:00200000:3.0:1788201563.014306:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xbb:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201563.014397:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xbb:0x0](ffff8e8289e46c08) for lock 0x9ae3109cee9e1ede 00000080:00010000:3.0:1788201563.014408:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xbb:0x0](ffff8e8289e46c08) 00000080:00010000:3.0:1788201563.014411:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ede from it ffffaa1802393e00 00010000:00010000:3.0:1788201563.014416:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1ede lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc192a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.014425:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1ede lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc192a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.014457:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201563.014474:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201563.014484:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ed7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc190e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.014494:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ed7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc190e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.014505:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201563.014509:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201563.014512:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1ed7 00000080:00010000:3.0:1788201563.014516:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ed7 from it ffffaa1802393d70 00010000:00010000:3.0:1788201563.014519:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ed7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc190e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.014525:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ed7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc190e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201563.014533:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201563.014540:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-72, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201563.014564:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ed7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc190e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201563.014587:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1ede lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xbb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc192a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.014608:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xbb:0x0]: losing layout lock 00010000:00010000:3.0:1788201563.014641:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1ede lrc: 1/0,0 mode: --/PR res: [0x200006991:0xbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc192a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.014648:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ed7 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc190e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201563.014654:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201563.014657:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1ede lrc: 0/0,0 mode: --/PR res: [0x200006991:0xbb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc192a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.014674:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1ed7 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc190e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201563.014691:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201563.014698:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828952aa00 x1875062187518080/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201563.014720:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952aa00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187518080:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201563.014813:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.019363:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952aa00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187518080:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201563.019426:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xbb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201563.019438:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201563.019466:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xbb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201563.019474:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xbb:0x0](ffff8e8289e46c08) 00000080:00200000:3.0:1788201563.019560:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201563.019582:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201563.019639:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ee5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.019649:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ee5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.019660:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ee5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201563.019679:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289528380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187518208:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201563.019797:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.023614:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289528380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187518208:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201563.023640:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ee5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1954 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.023652:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ee5 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1954 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.023662:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ee5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1954 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.023673:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ee5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1954 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.023685:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ee5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1954 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.023693:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ee5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1954 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.023700:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ee5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1954 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201563.023711:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289528380 x1875062187518208/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201563.023724:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ee5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1954 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.023742:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201563.023748:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201563.023783:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201563.023790:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1ee5 00000080:00010000:3.0:1788201563.023795:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201563.023798:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1ee5 from it ffffaa1802393c10 00010000:00010000:3.0:1788201563.023802:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ee5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1954 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.023810:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e1ee5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1954 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.023832:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201563.023853:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-73, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201563.023857:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-73, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201563.023876:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-73,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201563.023912:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1eec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.023919:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1eec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.023927:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1eec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201563.023944:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289528380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187518336:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201563.024124:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.033690:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289528380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187518336:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201563.033719:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e1eec lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1970 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:3.0:1788201563.108863:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-77, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201563.108872:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-77, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201563.108896:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-77,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201563.109078:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f24 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.109087:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f24 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.109096:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f24 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201563.109118:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289528380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187520384:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201563.109258:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.121259:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289528380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187520384:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201563.121287:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f24 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1a88 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.121428:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f24 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1a88 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.121437:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f24 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1a88 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.121456:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f24 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1a88 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.121467:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f24 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1a88 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.121483:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f24 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1a88 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.121491:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f24 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1a88 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.121498:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f24 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1a88 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201563.121512:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289528380 x1875062187520384/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201563.121525:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f24 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1a88 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201563.121540:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f24 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1a88 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.121552:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/192 (0 0) 00000080:00200000:3.0:1788201563.121561:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201563.121730:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xc0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201563.121742:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201563.121779:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xc0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201563.122353:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xc0:0x0](ffff8e8289c09148): rc = 0 00000080:00200000:3.0:1788201563.122374:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xc0:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201563.122380:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xc0:0x0](ffff8e8289c09148) for lock 0x9ae3109cee9e1f24 00000080:00010000:3.0:1788201563.122391:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xc0:0x0](ffff8e8289c09148) 00000080:00010000:3.0:1788201563.122395:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1f24 from it ffffaa1802393e00 00010000:00010000:3.0:1788201563.122399:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f24 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc1a88 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.122410:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f24 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc1a88 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.122441:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201563.122457:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201563.122468:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561a00/0x9ae3109cee9e1f1d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1a6c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.122479:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561a00/0x9ae3109cee9e1f1d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1a6c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.122486:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201563.122490:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201563.122493:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1f1d 00000080:00010000:3.0:1788201563.122498:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1f1d from it ffffaa1802393d70 00010000:00010000:3.0:1788201563.122501:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561a00/0x9ae3109cee9e1f1d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1a6c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.122514:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561a00/0x9ae3109cee9e1f1d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1a6c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201563.122523:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201563.122530:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-77, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201563.122555:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561a00/0x9ae3109cee9e1f1d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc1a6c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201563.122578:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f24 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc1a88 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.122587:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xc0:0x0]: losing layout lock 00010000:00010000:3.0:1788201563.122746:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f24 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1a88 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.122754:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561a00/0x9ae3109cee9e1f1d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1a6c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201563.122760:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201563.122772:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f24 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xc0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1a88 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.122790:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561a00/0x9ae3109cee9e1f1d lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1a6c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201563.122810:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201563.122835:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289528380 x1875062187520640/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201563.122860:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289528380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187520640:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201563.122951:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.127248:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289528380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187520640:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201563.127345:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xc0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201563.127359:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201563.127388:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xc0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201563.127396:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xc0:0x0](ffff8e8289c09148) 00000080:00200000:3.0:1788201563.127491:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201563.127512:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201563.127570:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f2b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.127581:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f2b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.127604:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f2b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201563.127625:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187520768:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201563.128019:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.130230:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187520768:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201563.130254:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f2b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1ab2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.130265:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f2b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1ab2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.130281:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f2b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1ab2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.130291:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f2b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1ab2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.130303:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f2b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1ab2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.130347:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f2b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1ab2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.130354:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f2b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1ab2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201563.130366:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289529880 x1875062187520768/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201563.130380:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f2b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1ab2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.130392:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201563.130399:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201563.130427:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201563.130433:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1f2b 00000080:00010000:3.0:1788201563.130438:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201563.130441:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1f2b from it ffffaa1802393c10 00010000:00010000:3.0:1788201563.130445:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f2b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1ab2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.130453:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f2b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1ab2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.130476:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201563.130498:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-78, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201563.130502:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-78, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201563.130521:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-78,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201563.130556:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f32 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.130563:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f32 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.130572:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f32 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201563.130590:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187520896:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201563.130662:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.134903:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187520896:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201563.134929:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f32 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1ace expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.134949:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f32 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1ace expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.134963:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f32 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1ace expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.134973:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f32 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1ace expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.134983:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f32 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1ace expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.135046:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f32 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1ace expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.135057:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f32 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1ace expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.135065:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f32 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1ace expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201563.135078:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289529880 x1875062187520896/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201563.135090:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f32 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1ace expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201563.135102:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f32 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1ace expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.135117:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/193 (0 0) 00000080:00200000:3.0:1788201563.135124:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201563.135161:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xc1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201563.135172:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201563.135195:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xc1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201563.135243:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xc1:0x0](ffff8e8289c0b248): rc = 0 00000080:00200000:3.0:1788201563.135256:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xc1:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201563.135261:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xc1:0x0](ffff8e8289c0b248) for lock 0x9ae3109cee9e1f32 00000080:00010000:3.0:1788201563.135271:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xc1:0x0](ffff8e8289c0b248) 00000080:00010000:3.0:1788201563.135274:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1f32 from it ffffaa1802393e00 00010000:00010000:3.0:1788201563.135277:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f32 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc1ace expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.135286:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f32 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc1ace expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.135881:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201563.135900:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201563.135910:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f2b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1ab2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.135921:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f2b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1ab2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.135928:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201563.135932:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201563.135936:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1f2b 00000080:00010000:3.0:1788201563.135939:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1f2b from it ffffaa1802393d70 00010000:00010000:3.0:1788201563.135943:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f2b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1ab2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.135950:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f2b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1ab2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201563.135959:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201563.135966:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-78, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201563.135991:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f2b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc1ab2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201563.136113:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f32 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc1ace expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.136123:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xc1:0x0]: losing layout lock 00010000:00010000:3.0:1788201563.136154:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f32 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1ace expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.136161:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f2b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1ab2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201563.136167:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201563.136170:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f32 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xc1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1ace expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.136186:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f2b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1ab2 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201563.136207:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201563.136215:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289529880 x1875062187521152/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201563.136236:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187521152:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201563.136397:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.151345:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187521152:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201563.254658:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201563.254683:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201563.254731:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f71 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.254738:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f71 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.254745:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f71 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201563.254759:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952a300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187523328:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201563.255357:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.257979:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952a300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187523328:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201563.258010:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f71 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c10 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.258021:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f71 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c10 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.258039:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f71 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c10 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.258052:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f71 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c10 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.258066:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f71 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c10 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.258074:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f71 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c10 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.258082:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f71 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c10 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201563.258094:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828952a300 x1875062187523328/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201563.258108:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f71 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c10 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.258120:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201563.258126:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201563.258154:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201563.258161:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1f71 00000080:00010000:3.0:1788201563.258166:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201563.258169:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1f71 from it ffffaa1802393c10 00010000:00010000:3.0:1788201563.258173:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f71 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c10 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.258181:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f71 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c10 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.258205:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201563.258229:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-83, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201563.258233:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-83, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201563.258253:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-83,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201563.258290:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f78 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.258295:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f78 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.258302:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f78 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201563.258349:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952a300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187523456:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201563.258416:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.261410:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952a300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187523456:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201563.261436:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f78 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c2c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.261457:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f78 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xc6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c2c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.261470:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f78 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xc6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c2c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.261480:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f78 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xc6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c2c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.261490:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f78 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xc6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c2c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.261501:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f78 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xc6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c2c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.261508:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f78 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xc6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c2c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.261516:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f78 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xc6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c2c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201563.261527:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828952a300 x1875062187523456/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201563.261540:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f78 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c2c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201563.261553:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f78 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c2c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.261567:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/198 (0 0) 00000080:00200000:3.0:1788201563.261573:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201563.261625:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xc6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201563.261637:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201563.261780:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xc6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201563.261955:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xc6:0x0](ffff8e8289e4ec08): rc = 0 00000080:00200000:3.0:1788201563.261972:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xc6:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201563.261977:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xc6:0x0](ffff8e8289e4ec08) for lock 0x9ae3109cee9e1f78 00000080:00010000:3.0:1788201563.261987:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xc6:0x0](ffff8e8289e4ec08) 00000080:00010000:3.0:1788201563.261990:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1f78 from it ffffaa1802393e00 00010000:00010000:3.0:1788201563.261994:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f78 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc1c2c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.262004:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f78 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xc6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc1c2c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.262034:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201563.262051:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201563.262061:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f71 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c10 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.262071:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f71 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c10 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.262077:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201563.262081:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201563.262085:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1f71 00000080:00010000:3.0:1788201563.262089:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1f71 from it ffffaa1802393d70 00010000:00010000:3.0:1788201563.262092:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f71 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c10 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.262099:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f71 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c10 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201563.262107:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201563.262113:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-83, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201563.262137:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f71 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc1c10 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201563.262156:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f78 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xc6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc1c2c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.262166:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xc6:0x0]: losing layout lock 00010000:00010000:3.0:1788201563.262194:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f78 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xc6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1c2c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.262201:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f71 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1c10 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201563.262207:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201563.262209:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f78 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xc6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1c2c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.262226:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f71 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1c10 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201563.262248:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201563.262256:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828952a300 x1875062187523712/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201563.262279:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952a300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187523712:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201563.262439:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.265498:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952a300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187523712:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201563.265546:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xc6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201563.265556:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201563.265580:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xc6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201563.265586:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xc6:0x0](ffff8e8289e4ec08) 00000080:00200000:3.0:1788201563.265670:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201563.265688:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201563.265742:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f7f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.265750:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f7f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.265756:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f7f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201563.265790:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187523840:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201563.266822:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.272296:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187523840:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201563.272358:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f7f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c56 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.272371:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f7f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c56 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.272383:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f7f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c56 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.272390:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f7f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c56 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.272400:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f7f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c56 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.272405:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f7f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c56 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.272409:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f7f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c56 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201563.272417:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289529500 x1875062187523840/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201563.272427:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f7f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c56 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.272435:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201563.272441:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201563.272465:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201563.272470:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1f7f 00000080:00010000:3.0:1788201563.272473:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201563.272476:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1f7f from it ffffaa1802393c10 00010000:00010000:3.0:1788201563.272478:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f7f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c56 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.272484:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f7f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c56 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.272504:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201563.272521:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-84, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201563.272525:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-84, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201563.272542:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-84,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201563.272570:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f86 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.272575:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f86 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.272581:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f86 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201563.272596:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187523968:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201563.272673:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.275289:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187523968:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201563.275403:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f86 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c72 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.275430:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f86 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xc7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c72 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.275438:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f86 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xc7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c72 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.275448:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f86 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xc7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c72 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.275456:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f86 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xc7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c72 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.275466:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f86 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xc7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c72 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.275471:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f86 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xc7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c72 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.275476:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f86 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xc7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c72 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201563.275484:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289529500 x1875062187523968/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201563.275492:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f86 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c72 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201563.275504:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f86 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c72 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.275511:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/199 (0 0) 00000080:00200000:3.0:1788201563.275517:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201563.275552:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xc7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201563.275561:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201563.275584:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xc7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201563.275636:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xc7:0x0](ffff8e8289e48908): rc = 0 00000080:00200000:3.0:1788201563.275651:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xc7:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201563.275657:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xc7:0x0](ffff8e8289e48908) for lock 0x9ae3109cee9e1f86 00000080:00010000:3.0:1788201563.275665:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xc7:0x0](ffff8e8289e48908) 00000080:00010000:3.0:1788201563.275668:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1f86 from it ffffaa1802393e00 00010000:00010000:3.0:1788201563.275671:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f86 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc1c72 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.275680:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f86 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xc7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc1c72 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.275708:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201563.275724:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201563.275731:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f7f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c56 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.275738:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f7f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c56 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.275746:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201563.275749:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201563.275752:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1f7f 00000080:00010000:3.0:1788201563.275754:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1f7f from it ffffaa1802393d70 00010000:00010000:3.0:1788201563.275756:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f7f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c56 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.275774:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f7f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c56 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201563.275783:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201563.275795:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-84, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201563.277548:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f7f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc1c56 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201563.277585:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f86 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xc7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc1c72 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.277597:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xc7:0x0]: losing layout lock 00010000:00010000:3.0:1788201563.277640:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f86 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xc7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1c72 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.277648:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f7f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1c56 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201563.277654:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201563.277657:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e1f86 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xc7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1c72 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.277676:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1f7f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1c56 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201563.277694:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201563.277704:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289529500 x1875062187524224/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201563.277728:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187524224:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201563.278007:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.282202:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187524224:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201563.282260:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xc7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201563.282273:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201563.282329:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xc7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201563.282338:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xc7:0x0](ffff8e8289e48908) 00000080:00200000:3.0:1788201563.282496:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201563.282518:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201563.282577:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f8d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.282657:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f8d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.282671:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f8d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201563.282693:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952bb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187524352:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201563.282805:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.285025:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952bb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187524352:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201563.285050:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f8d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c9c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.285059:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f8d lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c9c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.285068:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f8d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c9c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.285075:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f8d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c9c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.285083:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f8d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c9c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.285089:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f8d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c9c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.285093:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f8d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c9c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201563.285102:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828952bb80 x1875062187524352/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201563.285111:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f8d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c9c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.285123:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201563.285128:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201563.285149:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201563.285154:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1f8d 00000080:00010000:3.0:1788201563.285157:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201563.285160:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1f8d from it ffffaa1802393c10 00010000:00010000:3.0:1788201563.285162:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f8d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1c9c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.285167:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e1f8d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1c9c expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.285184:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201563.285200:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-85, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201563.285203:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-85, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201563.285218:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-85,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201563.285243:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e1f94 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.285248:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e1f94 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.285253:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e1f94 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201563.285266:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952bb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187524480:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201563.285538:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.289070:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952bb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187524480:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201563.289096:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e1f94 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1cb8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.289119:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e1f94 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xc8:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1cb8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.289127:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e1f94 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xc8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1cb8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.289137:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e1f94 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xc8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1cb8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.289147:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e1f94 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xc8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1cb8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.289158:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e1f94 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xc8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1cb8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.289166:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e1f94 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xc8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1cb8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.289174:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e1f94 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xc8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1cb8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201563.289185:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828952bb80 x1875062187524480/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201563.289196:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e1f94 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1cb8 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201563.289210:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e1f94 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xc8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1cb8 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.289219:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/200 (0 0) 00000080:00200000:3.0:1788201563.289226:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201563.289269:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xc8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201563.289279:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201563.289305:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xc8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201563.289448:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xc8:0x0](ffff8e8289e4f448): rc = 0 00000100:00100000:0.0:1788201563.321120:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289553b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187525376:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201563.321146:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e1fa9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d28 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.321158:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e1fa9 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d28 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.321169:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e1fa9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1d28 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.321180:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e1fa9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d28 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.321192:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e1fa9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1d28 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.321201:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e1fa9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1d28 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.321209:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e1fa9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1d28 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201563.321219:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289553b80 x1875062187525376/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201563.321233:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e1fa9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d28 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.321244:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201563.321250:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201563.321276:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201563.321281:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1fa9 00000080:00010000:0.0:1788201563.321286:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201563.321288:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1fa9 from it ffffaa1802393c10 00010000:00010000:0.0:1788201563.321292:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e1fa9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1d28 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.321301:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e1fa9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d28 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201563.322949:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201563.322980:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-87, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201563.322985:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-87, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201563.323121:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-87,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201563.323173:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.323181:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.323190:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201563.323214:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187525504:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201563.323324:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201563.331186:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187525504:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201563.331215:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d44 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.331238:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb0 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xca:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d44 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.331246:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb0 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d44 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.331257:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb0 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1d44 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.331267:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb0 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d44 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.331279:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb0 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1d44 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.331288:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb0 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1d44 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.331295:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb0 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1d44 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201563.331353:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793dc00 x1875062187525504/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201563.331366:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb0 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d44 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201563.331377:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb0 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d44 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.331387:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/202 (0 0) 00000080:00200000:0.0:1788201563.331393:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201563.331435:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xca:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201563.331445:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201563.331504:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xca:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201563.331563:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xca:0x0](ffff8e8289e40908): rc = 0 00000080:00200000:0.0:1788201563.331578:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xca:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201563.331583:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xca:0x0](ffff8e8289e40908) for lock 0x9ae3109cee9e1fb0 00000080:00010000:0.0:1788201563.331752:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xca:0x0](ffff8e8289e40908) 00000080:00010000:0.0:1788201563.331757:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1fb0 from it ffffaa1802393e00 00010000:00010000:0.0:1788201563.331770:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb0 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc1d44 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.331785:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb0 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc1d44 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201563.331818:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201563.331837:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201563.331847:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e1fa9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d28 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201563.331857:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e1fa9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d28 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201563.331864:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201563.331868:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201563.331871:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1fa9 00000080:00010000:0.0:1788201563.331875:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1fa9 from it ffffaa1802393d70 00010000:00010000:0.0:1788201563.331878:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e1fa9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1d28 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201563.331886:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e1fa9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d28 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201563.331894:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201563.331901:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-87, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201563.331926:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e1fa9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc1d28 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201563.331948:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb0 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xca:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc1d44 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201563.331958:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xca:0x0]: losing layout lock 00010000:00010000:0.0:1788201563.331987:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb0 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1d44 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201563.332058:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e1fa9 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1d28 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201563.332066:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201563.332069:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb0 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xca:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1d44 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201563.332086:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e1fa9 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1d28 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201563.332106:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201563.332115:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793dc00 x1875062187525760/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201563.332137:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187525760:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201563.332262:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00010000:00010000:3.0:1788201563.336533:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201563.341701:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187525760:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201563.341770:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xca:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201563.341784:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201563.341812:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xca:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201563.341821:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xca:0x0](ffff8e8289e40908) 00000080:00200000:0.0:1788201563.341908:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201563.341929:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201563.341985:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.342042:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.342055:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201563.342076:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187525888:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201563.342199:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201563.344563:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187525888:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201563.344652:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d6e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.344726:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb7 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d6e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.344740:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1d6e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.344752:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d6e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.344775:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1d6e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.344784:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1d6e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.344792:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1d6e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201563.344805:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793c700 x1875062187525888/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201563.344820:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d6e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.344834:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201563.344841:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201563.344866:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201563.344873:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1fb7 00000080:00010000:0.0:1788201563.344878:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201563.344882:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1fb7 from it ffffaa1802393c10 00010000:00010000:0.0:1788201563.344886:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1d6e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.344895:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d6e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201563.344919:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201563.344948:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-88, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201563.344953:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-88, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201563.344973:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-88,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201563.345141:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1fbe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.345150:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1fbe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.345159:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1fbe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201563.345178:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187526016:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201563.345298:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201563.347985:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187526016:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201563.348277:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1fbe lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d8a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.348300:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1fbe lrc: 4/1,0 mode: --/PR res: [0x200006991:0xcb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d8a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.348366:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1fbe lrc: 4/1,0 mode: --/PR res: [0x200006991:0xcb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d8a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.348377:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1fbe lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xcb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1d8a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.348388:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1fbe lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xcb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d8a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.348399:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1fbe lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xcb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1d8a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.348412:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1fbe lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xcb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1d8a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.348419:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1fbe lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xcb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1d8a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201563.348432:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793c700 x1875062187526016/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201563.348444:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1fbe lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xcb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d8a expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201563.348458:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1fbe lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xcb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d8a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.348468:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/203 (0 0) 00000080:00200000:0.0:1788201563.348475:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201563.348514:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xcb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201563.348525:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201563.348552:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xcb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201563.348747:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xcb:0x0](ffff8e8289e4a1c8): rc = 0 00000080:00200000:0.0:1788201563.348773:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xcb:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201563.348779:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xcb:0x0](ffff8e8289e4a1c8) for lock 0x9ae3109cee9e1fbe 00000080:00010000:0.0:1788201563.348788:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xcb:0x0](ffff8e8289e4a1c8) 00000080:00010000:0.0:1788201563.348792:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1fbe from it ffffaa1802393e00 00010000:00010000:0.0:1788201563.348796:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1fbe lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xcb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc1d8a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.348811:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1fbe lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xcb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc1d8a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201563.348841:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201563.348856:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201563.348865:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d6e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201563.348876:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d6e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201563.348882:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201563.348886:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201563.348889:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1fb7 00000080:00010000:0.0:1788201563.348892:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1fb7 from it ffffaa1802393d70 00010000:00010000:0.0:1788201563.348895:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1d6e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201563.348902:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1d6e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201563.348910:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201563.348916:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-88, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201563.348939:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc1d6e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201563.348960:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1fbe lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xcb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc1d8a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201563.348969:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xcb:0x0]: losing layout lock 00010000:00010000:0.0:1788201563.349055:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1fbe lrc: 1/0,0 mode: --/PR res: [0x200006991:0xcb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1d8a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201563.349064:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb7 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1d6e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201563.349070:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201563.349073:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0e00/0x9ae3109cee9e1fbe lrc: 0/0,0 mode: --/PR res: [0x200006991:0xcb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1d8a expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201563.349089:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fb7 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1d6e expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201563.349106:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201563.349114:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793c700 x1875062187526272/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201563.349135:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187526272:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201563.349260:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201563.356256:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187526272:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201563.356370:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xcb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201563.356383:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201563.356412:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xcb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201563.356419:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xcb:0x0](ffff8e8289e4a1c8) 00000080:00200000:0.0:1788201563.356511:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201563.356530:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201563.356578:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1fc5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.356773:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1fc5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.356786:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1fc5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201563.356807:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187526400:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201563.356938:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201563.359956:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187526400:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201563.359980:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1fc5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1db4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.359991:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1fc5 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1db4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.360083:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1fc5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1db4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.360094:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1fc5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1db4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.360108:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1fc5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1db4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.360116:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1fc5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1db4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.360123:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1fc5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1db4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201563.360135:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793f800 x1875062187526400/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201563.360149:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1fc5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1db4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.360161:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201563.360168:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201563.360195:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201563.360200:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1fc5 00000080:00010000:0.0:1788201563.360206:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201563.360209:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1fc5 from it ffffaa1802393c10 00010000:00010000:0.0:1788201563.360212:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1fc5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1db4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.360220:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1fc5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1db4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201563.360243:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201563.360263:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-89, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201563.360267:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-89, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201563.360285:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-89,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201563.360521:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fcc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.360529:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fcc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.360539:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fcc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201563.360558:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187526528:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201563.360642:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201563.363188:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187526528:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201563.363213:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fcc lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1dd0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.363234:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fcc lrc: 4/1,0 mode: --/PR res: [0x200006991:0xcc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1dd0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.363243:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fcc lrc: 4/1,0 mode: --/PR res: [0x200006991:0xcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1dd0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.363254:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fcc lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1dd0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.363265:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fcc lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1dd0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.363277:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fcc lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1dd0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.363291:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fcc lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1dd0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.363299:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fcc lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1dd0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201563.363331:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793f800 x1875062187526528/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201563.363343:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fcc lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1dd0 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201563.363355:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fcc lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1dd0 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.363365:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/204 (0 0) 00000080:00200000:0.0:1788201563.363371:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201563.363411:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xcc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201563.363423:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201563.363450:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xcc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201563.363506:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xcc:0x0](ffff8e8289e580c8): rc = 0 00000080:00200000:0.0:1788201563.363522:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xcc:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201563.363528:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xcc:0x0](ffff8e8289e580c8) for lock 0x9ae3109cee9e1fcc 00000080:00010000:0.0:1788201563.363538:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xcc:0x0](ffff8e8289e580c8) 00000080:00010000:0.0:1788201563.363541:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1fcc from it ffffaa1802393e00 00010000:00010000:0.0:1788201563.363546:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fcc lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc1dd0 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.363560:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fcc lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc1dd0 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201563.363841:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201563.364015:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201563.364029:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1fc5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1db4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201563.364041:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1fc5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1db4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201563.364047:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201563.364052:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201563.364055:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1fc5 00000080:00010000:1.0:1788201563.364058:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1fc5 from it ffffaa1802393d70 00010000:00010000:1.0:1788201563.364061:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1fc5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1db4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201563.364068:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1fc5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1db4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201563.364077:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201563.364084:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-89, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201563.364110:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1fc5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc1db4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201563.364132:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fcc lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xcc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc1dd0 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201563.364141:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xcc:0x0]: losing layout lock 00010000:00010000:1.0:1788201563.364172:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fcc lrc: 1/0,0 mode: --/PR res: [0x200006991:0xcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1dd0 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201563.364180:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1fc5 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1db4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201563.364187:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201563.364190:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0600/0x9ae3109cee9e1fcc lrc: 0/0,0 mode: --/PR res: [0x200006991:0xcc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1dd0 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201563.364205:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0200/0x9ae3109cee9e1fc5 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1db4 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201563.364223:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201563.364232:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289510000 x1875062187526784/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201563.364254:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510000 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187526784:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201563.364368:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.367709:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510000 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187526784:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201563.367780:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xcc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201563.367793:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201563.367821:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xcc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201563.367829:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xcc:0x0](ffff8e8289e580c8) 00000080:00200000:3.0:1788201563.367922:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201563.367943:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201563.368211:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1fd3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.368223:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1fd3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.368234:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1fd3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201563.368255:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529c00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187526912:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201563.368546:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.371260:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529c00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187526912:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201563.371284:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1fd3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1dfa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.371297:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1fd3 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1dfa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.371449:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1fd3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1dfa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.371463:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1fd3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1dfa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.371477:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1fd3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1dfa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.371484:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1fd3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1dfa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.371491:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1fd3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1dfa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201563.371504:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289529c00 x1875062187526912/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201563.371518:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1fd3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1dfa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.371533:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201563.371540:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201563.371741:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201563.371750:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1fd3 00000080:00010000:0.0:1788201563.371755:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201563.371769:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1fd3 from it ffffaa1802393c10 00010000:00010000:0.0:1788201563.371774:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1fd3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1dfa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.371783:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1fd3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1dfa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201563.371806:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201563.371829:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-90, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201563.371834:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-90, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201563.371857:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-90,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201563.371898:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1fda lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.371906:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1fda lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.371914:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1fda lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201563.371934:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187527040:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201563.372114:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201563.375487:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187527040:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201563.375511:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1fda lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1e16 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.375532:0:41526:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1fda lrc: 4/1,0 mode: --/PR res: [0x200006991:0xcd:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1e16 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.375540:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1fda lrc: 4/1,0 mode: --/PR res: [0x200006991:0xcd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1e16 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.375551:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1fda lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xcd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1e16 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.375562:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1fda lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xcd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1e16 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.375575:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1fda lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xcd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1e16 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.375583:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1fda lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xcd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1e16 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.375602:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1fda lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xcd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1e16 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201563.375613:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793f800 x1875062187527040/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201563.375626:0:41526:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1fda lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xcd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1e16 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201563.375638:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1fda lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xcd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1e16 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.375648:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/205 (0 0) 00000080:00200000:0.0:1788201563.375655:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201563.375703:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xcd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201563.375714:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201563.375737:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xcd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201563.375847:0:41526:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xcd:0x0](ffff8e8289e5aa08): rc = 0 00000080:00200000:0.0:1788201563.375863:0:41526:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xcd:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201563.375869:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xcd:0x0](ffff8e8289e5aa08) for lock 0x9ae3109cee9e1fda 00000080:00010000:0.0:1788201563.375878:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xcd:0x0](ffff8e8289e5aa08) 00000080:00010000:0.0:1788201563.375882:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1fda from it ffffaa1802393e00 00010000:00010000:0.0:1788201563.375886:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1fda lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xcd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc1e16 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.375894:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1fda lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xcd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc1e16 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201563.375927:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201563.375943:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201563.375953:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1fd3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1dfa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201563.375968:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1fd3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1dfa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201563.375975:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201563.375979:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201563.375982:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1fd3 00000080:00010000:0.0:1788201563.375986:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1fd3 from it ffffaa1802393d70 00010000:00010000:0.0:1788201563.375988:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1fd3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1dfa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201563.376178:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1fd3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1dfa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201563.376191:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201563.376199:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-90, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201563.376225:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1fd3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc1dfa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201563.376247:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1fda lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xcd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc1e16 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201563.376256:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xcd:0x0]: losing layout lock 00010000:00010000:0.0:1788201563.376285:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1fda lrc: 1/0,0 mode: --/PR res: [0x200006991:0xcd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1e16 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201563.376293:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1fd3 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1dfa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201563.376304:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201563.376460:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0400/0x9ae3109cee9e1fda lrc: 0/0,0 mode: --/PR res: [0x200006991:0xcd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc1e16 expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201563.376479:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559c00/0x9ae3109cee9e1fd3 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc1dfa expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201563.376500:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201563.376508:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793f800 x1875062187527296/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201563.376532:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187527296:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201563.376621:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201563.380143:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187527296:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201563.380202:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xcd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201563.380215:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201563.380241:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xcd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201563.380248:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xcd:0x0](ffff8e8289e5aa08) 00000080:00200000:0.0:1788201563.380366:0:41526:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201563.380386:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201563.380441:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1000/0x9ae3109cee9e1fe1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.380450:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1000/0x9ae3109cee9e1fe1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.380460:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1000/0x9ae3109cee9e1fe1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201563.380480:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187527424:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201563.380549:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:0.0:1788201563.383172:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187527424:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201563.383198:0:41526:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1000/0x9ae3109cee9e1fe1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1e40 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.383210:0:41526:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1000/0x9ae3109cee9e1fe1 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1e40 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.383221:0:41526:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1000/0x9ae3109cee9e1fe1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1e40 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.383232:0:41526:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1000/0x9ae3109cee9e1fe1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1e40 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.383242:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1000/0x9ae3109cee9e1fe1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1e40 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.383250:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1000/0x9ae3109cee9e1fe1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1e40 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.383257:0:41526:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1000/0x9ae3109cee9e1fe1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1e40 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201563.383270:0:41526:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793ce00 x1875062187527424/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201563.383290:0:41526:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1000/0x9ae3109cee9e1fe1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1e40 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.383302:0:41526:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201563.383507:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201563.383727:0:41526:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201563.383737:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e1fe1 00000080:00010000:1.0:1788201563.383744:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201563.383747:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e1fe1 from it ffffaa1802393c10 00010000:00010000:1.0:1788201563.383750:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1000/0x9ae3109cee9e1fe1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc1e40 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201563.383769:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1000/0x9ae3109cee9e1fe1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc1e40 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201563.383797:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201563.383819:0:41526:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f11b.replay-vbr-1-91, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201563.383825:0:41526:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f11b.replay-vbr-1-91, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201563.383848:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f11b.replay-vbr-1-91,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201563.383893:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e1fe8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201563.383900:0:41526:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e1fe8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201563.383909:0:41526:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e1fe8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 41526 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201563.383930:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187527552:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201563.384173:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1240 going to sleep for 11 seconds 00000002:00010000:0.0:1788201563.467516:0:41526:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201563.467526:0:41526:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561800/0x9ae3109cee9e2051 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2070 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201563.467533:0:41526:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561800/0x9ae3109cee9e2051 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2070 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201563.467538:0:41526:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201563.467541:0:41526:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201563.467543:0:41526:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2051 00000080:00010000:0.0:1788201563.467546:0:41526:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2051 from it ffffaa1802393d70 00010000:00010000:0.0:1788201563.467548:0:41526:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561800/0x9ae3109cee9e2051 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2070 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201563.467552:0:41526:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561800/0x9ae3109cee9e2051 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2070 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201563.467558:0:41526:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201563.467563:0:41526:0:(namei.c:2418:ll_unlink()) VFS Op:name=f11b.replay-vbr-1-99, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201563.467585:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561800/0x9ae3109cee9e2051 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc2070 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201563.467612:0:41526:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561200/0x9ae3109cee9e2058 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xd6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc208c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201563.467622:0:41526:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xd6:0x0]: losing layout lock 00010000:00010000:0.0:1788201563.467655:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561200/0x9ae3109cee9e2058 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc208c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201563.467720:0:41526:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561800/0x9ae3109cee9e2051 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2070 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201563.467726:0:41526:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201563.467729:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561200/0x9ae3109cee9e2058 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xd6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc208c expref: -99 pid: 41526 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201563.467749:0:41526:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561800/0x9ae3109cee9e2051 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2070 expref: -99 pid: 41526 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201563.467777:0:41526:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201563.467786:0:41526:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793d180 x1875062187531904/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201563.467806:0:41526:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187531904:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201563.467916:0:41526:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a000 going to sleep for 11 seconds 00000100:00100000:1.0:1788201563.471187:0:41526:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:41526:1875062187531904:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201563.471256:0:41526:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xd6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201563.471269:0:41526:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201563.471299:0:41526:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xd6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201563.471453:0:41526:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xd6:0x0](ffff8e8289c080c8) 00000080:00200000:0.0:1788201563.480989:0:40852:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201563.481021:0:40852:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201563.481083:0:40852:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 40852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.481100:0:40852:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 40852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201563.481110:0:40852:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 40852 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201563.481135:0:40852:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job bash:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40852:1875062187532032:192.168.201.121@tcp:101:bash.0 00000100:00100000:0.0:1788201563.481278:0:40852:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828133a900 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.485151:0:40852:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job bash:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40852:1875062187532032:192.168.201.121@tcp:101:bash.0 00010000:00010000:3.0:1788201563.485179:0:40852:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.485192:0:40852:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.485203:0:40852:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.485240:0:40852:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.485255:0:40852:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.485264:0:40852:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.485271:0:40852:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201563.485284:0:40852:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793ce00 x1875062187532032/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'bash.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201563.485304:0:40852:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.485337:0:40852:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201563.485344:0:40852:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201563.485372:0:40852:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201563.485379:0:40852:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e205f 00000080:00010000:3.0:1788201563.485385:0:40852:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201563.485388:0:40852:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e205f from it ffffaa1801c97b58 00010000:00010000:3.0:1788201563.485392:0:40852:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.485401:0:40852:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201563.485425:0:40852:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201563.485444:0:40852:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d11b.replay-vbr, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=25 00000080:00200000:3.0:1788201563.485449:0:40852:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d11b.replay-vbr, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=0 00000002:00010000:3.0:1788201563.485470:0:40852:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d11b.replay-vbr,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1788201563.485513:0:40852:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e2066 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 40852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.485520:0:40852:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e2066 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.485530:0:40852:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e2066 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40852 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201563.485553:0:40852:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529c00 pname:cluuid:pid:xid:nid:opc:job bash:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40852:1875062187532160:192.168.201.121@tcp:101:bash.0 00000100:00100000:3.0:1788201563.485644:0:40852:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799ccc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201563.488152:0:40852:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529c00 pname:cluuid:pid:xid:nid:opc:job bash:dc31b02e-cf70-4b83-9f70-d7124af83e4d:40852:1875062187532160:192.168.201.121@tcp:101:bash.0 00010000:00010000:3.0:1788201563.488174:0:40852:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e2066 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 40852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.488185:0:40852:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e2066 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 40852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.488192:0:40852:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e2066 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 40852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.488198:0:40852:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e2066 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 40852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.488207:0:40852:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e2066 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 40852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.488213:0:40852:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8e8289559a00 already unused, calling callback (ffffffffc1c51460) 00010000:00010000:3.0:1788201563.488217:0:40852:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e2066 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 40852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.488224:0:40852:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e2066 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 40852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.488231:0:40852:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e2066 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 40852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.488237:0:40852:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788201563.488243:0:40852:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e2066 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 40852 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201563.488250:0:40852:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559a00/0x9ae3109cee9e2066 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 40852 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201563.488268:0:40852:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0x7, status=0 req@ffff8e8289529c00 x1875062187532160/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 584/584 e 0 to 0 dl 1788201579 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'bash.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201563.488280:0:40852:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1788201563.488304:0:40852:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.488345:0:40852:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788201563.488354:0:40852:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e205f from it ffffaa1801c978d0 00010000:00010000:3.0:1788201563.488358:0:40852:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201563.488366:0:40852:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201564.361359:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201565.380570:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201566.409990:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201566.745869:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289552680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2403:1875062187532288:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201566.745891:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187532544:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201566.752286:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289551180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187532416:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201566.752336:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289553b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187532672:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201566.755992:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187532800:192.168.201.121@tcp:400:kworker.0 00010000:00010000:3.0:1788201567.430802:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201568.453056:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201569.489870:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201570.500921:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000020:01000004:0.0:1788201570.504796:0:42118:0:(obd_mount.c:923:lmd_print()) mount data: 00000020:01000004:0.0:1788201570.504801:0:42118:0:(obd_mount.c:925:lmd_print()) profile: lustre-client 00000020:01000004:0.0:1788201570.504802:0:42118:0:(obd_mount.c:926:lmd_print()) device: 192.168.201.121@tcp:/lustre 00000020:01000004:0.0:1788201570.504804:0:42118:0:(obd_mount.c:929:lmd_print()) options: user_xattr,flock 00000080:01200004:0.0:1788201570.505054:0:42118:0:(super25.c:131:lustre_fill_super()) VFS Op: sb ffff8e82879e2000 00000080:01000004:0.0:1788201570.505064:0:42118:0:(super25.c:162:lustre_fill_super()) Mounting client lustre-client 00000020:01000004:0.0:1788201570.505086:0:42118:0:(obd_mount.c:403:lustre_start_mgc()) MGC192.168.201.121@tcp:Set MGC reconnect 1 10000000:01000000:0.0:1788201570.505089:0:42118:0:(mgc_request.c:1071:mgc_set_info_async()) InitRecov MGC192.168.201.121@tcp 1/d0:i0:r0:or0:FULL 00000080:00200000:0.0:1788201570.505097:0:42118:0:(llite_lib.c:1506:ll_fill_super()) VFS Op: cfg_instance lustre-client-ffff8e82879e2000 (sb ffff8e82879e2000) 00000080:01000000:0.0:1788201570.505173:0:42118:0:(llite_lib.c:1232:ll_options()) Parsing opts user_xattr,flock 00000080:00000004:0.0:1788201570.505175:0:42118:0:(llite_lib.c:1242:ll_options()) next opt=user_xattr 00000080:00000004:0.0:1788201570.505179:0:42118:0:(llite_lib.c:1242:ll_options()) next opt=flock 00000080:01000000:0.0:1788201570.505186:0:42118:0:(llite_lib.c:1540:ll_fill_super()) llite sb uuid: eaf4444c-5340-4caf-8764-d8e0f2a45321 10000000:01000000:0.0:1788201570.507376:0:42118:0:(mgc_request.c:1944:mgc_process_config()) parse_log lustre-client from 0 10000000:01000000:0.0:1788201570.507382:0:42118:0:(mgc_request.c:329:config_log_add()) add config log lustre-client-ffff8e82879e2000 10000000:01000000:0.0:1788201570.507389:0:42118:0:(mgc_request.c:194:do_config_log_add()) do adding config log params-ffff8e82879e2000 10000000:01000000:0.0:1788201570.507394:0:42118:0:(mgc_request.c:68:mgc_name2resid()) log params to resid 0x736d61726170/0x3 (params) 10000000:01000000:0.0:1788201570.507399:0:42118:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-client-ffff8e82879e2000 10000000:01000000:0.0:1788201570.507402:0:42118:0:(mgc_request.c:68:mgc_name2resid()) log lustre-client to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:0.0:1788201570.507405:0:42118:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-cliir-ffff8e82879e2000 10000000:01000000:0.0:1788201570.507408:0:42118:0:(mgc_request.c:68:mgc_name2resid()) log lustre-cliir to resid 0x65727473756c/0x2 (lustre) 10000000:01000000:0.0:1788201570.507412:0:42118:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff8e82879e2000 from 1 10000000:01000000:0.0:1788201570.507415:0:42118:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:0.0:1788201570.507453:0:42118:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1a00/0x9ae3109cee9e206d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201570.507823:0:42118:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1a00/0x9ae3109cee9e206d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201570.508281:0:42118:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1a00/0x9ae3109cee9e206d lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201570.508335:0:42118:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:42118:1875062187532928:192.168.201.121@tcp:101:mount.lustre.0 00000100:00100000:0.0:1788201570.508430:0:42118:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230d80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201570.510483:0:42118:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:42118:1875062187532928:192.168.201.121@tcp:101:mount.lustre.0 00010000:00010000:0.0:1788201570.510508:0:42118:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1a00/0x9ae3109cee9e206d lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc238e expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201570.510521:0:42118:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1a00/0x9ae3109cee9e206d lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1000000000000 nid: local remote: 0x7cf2114380cc238e expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201570.510591:0:42118:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:42118:1875062187533056:192.168.201.121@tcp:501:mount.lustre.0 00000100:00100000:0.0:1788201570.510661:0:42118:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230d80 going to sleep for 11 seconds 00000100:00100000:0.0:1788201570.518194:0:42118:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:42118:1875062187533056:192.168.201.121@tcp:501:mount.lustre.0 00000100:00100000:0.0:1788201570.518256:0:42118:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:42118:1875062187533184:192.168.201.121@tcp:503:mount.lustre.0 00000100:00100000:0.0:1788201570.518357:0:42118:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230d80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201570.543952:0:42122:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529c00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:42122:1875062187533312:192.168.201.121@tcp:502:llog_process_th.0 00000100:00100000:3.0:1788201570.544116:0:42122:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201570.572305:0:42122:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529c00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:42122:1875062187533312:192.168.201.121@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788201570.572385:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788201570.572397:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201570.572399:0:42122:0:(obd_config.c:1563:class_process_config()) marker 4 (0x1) lustre-clilov lov setup 00000020:01000000:3.0:1788201570.572449:0:42122:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilov-ffff8e82879e2000 00000020:00000080:3.0:1788201570.572453:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:3.0:1788201570.572467:0:42122:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilov-ffff8e82879e2000 (ffff8e8291bd99b8) 00000020:00000080:3.0:1788201570.572486:0:42122:0:(obd_config.c:720:class_attach_name()) OBD: dev 7 attached type lov with refcount 2 00000020:01000000:3.0:1788201570.572493:0:42122:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilov-ffff8e82879e2000 00000020:00000080:3.0:1788201570.572496:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:3.0:1788201570.572823:0:42122:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilov-ffff8e82879e2000 (uuid eaf4444c-5340-4caf-8764-d8e0f2a45321) 00000020:01000000:3.0:1788201570.572829:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788201570.572833:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201570.572835:0:42122:0:(obd_config.c:1563:class_process_config()) marker 4 (0x2) lustre-clilov lov setup 00000020:01000000:3.0:1788201570.572837:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788201570.572841:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201570.572842:0:42122:0:(obd_config.c:1563:class_process_config()) marker 5 (0x1) lustre-clilmv lmv setup 00000020:01000000:3.0:1788201570.572845:0:42122:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-clilmv-ffff8e82879e2000 00000020:00000080:3.0:1788201570.572847:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:3.0:1788201570.572854:0:42122:0:(genops.c:413:class_newdev()) Allocate new device lustre-clilmv-ffff8e82879e2000 (ffff8e82856fcd28) 00000020:00000080:3.0:1788201570.572864:0:42122:0:(obd_config.c:720:class_attach_name()) OBD: dev 8 attached type lmv with refcount 2 00000020:01000000:3.0:1788201570.572868:0:42122:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-clilmv-ffff8e82879e2000 00000020:00000080:3.0:1788201570.572871:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00000020:00000080:3.0:1788201570.573045:0:42122:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-clilmv-ffff8e82879e2000 (uuid eaf4444c-5340-4caf-8764-d8e0f2a45321) 00000020:01000000:3.0:1788201570.573050:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788201570.573053:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201570.573055:0:42122:0:(obd_config.c:1563:class_process_config()) marker 5 (0x2) lustre-clilmv lmv setup 00000020:01000000:3.0:1788201570.573057:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788201570.573060:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201570.573061:0:42122:0:(obd_config.c:1563:class_process_config()) marker 6 (0x1) lustre-client mount opts 00000020:00000080:3.0:1788201570.573064:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf007 00000020:00000080:3.0:1788201570.573066:0:42122:0:(obd_config.c:1516:class_process_config()) mountopt: profile lustre-client osc lustre-clilov mdc lustre-clilmv 00000020:01000000:3.0:1788201570.573069:0:42122:0:(obd_config.c:1149:class_add_profile()) Add profile lustre-client 00000020:01000000:3.0:1788201570.573073:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788201570.573075:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201570.573076:0:42122:0:(obd_config.c:1563:class_process_config()) marker 6 (0x2) lustre-client mount opts 00000020:01000000:3.0:1788201570.573079:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788201570.573082:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201570.573083:0:42122:0:(obd_config.c:1563:class_process_config()) marker 7 (0x1) lustre-MDT0000 add mdc 00000020:00000080:3.0:1788201570.573085:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:3.0:1788201570.573090:0:42122:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.201.121@tcp to nid 0x20000c0a8c979 (192.168.201.121@tcp) 00000020:01000000:3.0:1788201570.573110:0:42122:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0000-mdc-ffff8e82879e2000 00000020:00000080:3.0:1788201570.573112:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:3.0:1788201570.573119:0:42122:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0000-mdc-ffff8e82879e2000 (ffff8e82856f8000) 00000020:00000080:3.0:1788201570.573128:0:42122:0:(obd_config.c:720:class_attach_name()) OBD: dev 9 attached type mdc with refcount 2 00000020:01000000:3.0:1788201570.573133:0:42122:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0000-mdc-ffff8e82879e2000 00000020:00000080:3.0:1788201570.573135:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:3.0:1788201570.573191:0:42122:0:(ldlm_lib.c:102:import_set_conn()) imp ffff8e82911ce000@lustre-MDT0000-mdc-ffff8e82879e2000: add connection 192.168.201.121@tcp at head 00010000:00010000:3.0:1788201570.574832:0:42122:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0000-mdc-ffff8e82879e2000-5 is initialized 00000040:01000000:3.0:1788201570.575031:0:42122:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0000-mdc-ffff8e82879e2000 ctxt 13 is initialized 00000020:00000080:3.0:1788201570.575063:0:42122:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0000-mdc-ffff8e82879e2000 (uuid eaf4444c-5340-4caf-8764-d8e0f2a45321) 00000020:01000000:3.0:1788201570.575072:0:42122:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff8e82879e2000 00000020:00000080:3.0:1788201570.575075:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:3.0:1788201570.575092:0:42122:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0000_UUID. index 0 00000020:01000000:3.0:1788201570.575127:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788201570.575131:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201570.575132:0:42122:0:(obd_config.c:1563:class_process_config()) marker 7 (0x2) lustre-MDT0000 add mdc 00000020:01000000:3.0:1788201570.575135:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:3.0:1788201570.575138:0:42122:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #9 00000020:01000000:3.0:1788201570.575139:0:42122:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:3.0:1788201570.575141:0:42122:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:3.0:1788201570.575143:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:3.0:1788201570.575145:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201570.575146:0:42122:0:(obd_config.c:1563:class_process_config()) marker 9 (0x6) lustre sys.timeout 00000020:01000000:3.0:1788201570.575149:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788201570.575154:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201570.575155:0:42122:0:(obd_config.c:1563:class_process_config()) marker 16 (0x1) lustre-MDT0001 add mdc 00000020:00000080:3.0:1788201570.575158:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:3.0:1788201570.575162:0:42122:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.201.121@tcp to nid 0x20000c0a8c979 (192.168.201.121@tcp) 00000020:01000000:3.0:1788201570.575184:0:42122:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-MDT0001-mdc-ffff8e82879e2000 00000020:00000080:3.0:1788201570.575186:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:3.0:1788201570.575196:0:42122:0:(genops.c:413:class_newdev()) Allocate new device lustre-MDT0001-mdc-ffff8e82879e2000 (ffff8e82856f99b8) 00000020:00000080:3.0:1788201570.575207:0:42122:0:(obd_config.c:720:class_attach_name()) OBD: dev 10 attached type mdc with refcount 2 00000020:01000000:3.0:1788201570.575211:0:42122:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-MDT0001-mdc-ffff8e82879e2000 00000020:00000080:3.0:1788201570.575213:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:3.0:1788201570.575273:0:42122:0:(ldlm_lib.c:102:import_set_conn()) imp ffff8e82911c8000@lustre-MDT0001-mdc-ffff8e82879e2000: add connection 192.168.201.121@tcp at head 00010000:00010000:3.0:1788201570.586791:0:42122:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-MDT0001-mdc-ffff8e82879e2000-6 is initialized 00000040:01000000:3.0:1788201570.586998:0:42122:0:(llog_obd.c:197:llog_setup()) obd lustre-MDT0001-mdc-ffff8e82879e2000 ctxt 13 is initialized 00000020:00000080:3.0:1788201570.587042:0:42122:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-MDT0001-mdc-ffff8e82879e2000 (uuid eaf4444c-5340-4caf-8764-d8e0f2a45321) 00000020:01000000:3.0:1788201570.587057:0:42122:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf014, instance name: lustre-clilmv-ffff8e82879e2000 00000020:00000080:3.0:1788201570.587061:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf014 00800000:01000000:3.0:1788201570.587085:0:42122:0:(lmv_obd.c:442:lmv_add_target()) Target uuid: lustre-MDT0001_UUID. index 1 00000020:01000000:3.0:1788201570.587119:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788201570.587123:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201570.587124:0:42122:0:(obd_config.c:1563:class_process_config()) marker 16 (0x2) lustre-MDT0001 add mdc 00000020:01000000:3.0:1788201570.587127:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:3.0:1788201570.587130:0:42122:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #20 00000020:01000000:3.0:1788201570.587132:0:42122:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:3.0:1788201570.587134:0:42122:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:3.0:1788201570.587135:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:3.0:1788201570.587137:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201570.587138:0:42122:0:(obd_config.c:1563:class_process_config()) marker 20 (0x6) lustre sys.timeout 00000020:01000000:3.0:1788201570.587141:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:3.0:1788201570.587145:0:42122:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0000 (0) in 0 of 192.168.201.121@tcp:/lustre 00000020:00000080:3.0:1788201570.587148:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201570.587149:0:42122:0:(obd_config.c:1563:class_process_config()) marker 24 (0x1) lustre-OST0000 add osc 00000020:00000080:3.0:1788201570.587152:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:3.0:1788201570.587158:0:42122:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.201.121@tcp to nid 0x20000c0a8c979 (192.168.201.121@tcp) 00000020:01000000:3.0:1788201570.587183:0:42122:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0000-osc-ffff8e82879e2000 00000020:00000080:3.0:1788201570.587186:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:3.0:1788201570.587196:0:42122:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0000-osc-ffff8e82879e2000 (ffff8e82856fb370) 00000020:00000080:3.0:1788201570.587209:0:42122:0:(obd_config.c:720:class_attach_name()) OBD: dev 11 attached type osc with refcount 2 00000020:01000000:3.0:1788201570.587214:0:42122:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0000-osc-ffff8e82879e2000 00000020:00000080:3.0:1788201570.587216:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:3.0:1788201570.587275:0:42122:0:(ldlm_lib.c:102:import_set_conn()) imp ffff8e82911c9000@lustre-OST0000-osc-ffff8e82879e2000: add connection 192.168.201.121@tcp at head 00010000:00010000:3.0:1788201570.587443:0:42122:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0000-osc-ffff8e82879e2000-7 is initialized 00000020:00000080:3.0:1788201570.587645:0:42122:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0000-osc-ffff8e82879e2000 (uuid eaf4444c-5340-4caf-8764-d8e0f2a45321) 00000020:01000000:3.0:1788201570.587652:0:42122:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff8e82879e2000 00000020:00000080:3.0:1788201570.587655:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:3.0:1788201570.587666:0:42122:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0000_UUID idx:0 gen:1 active:1 00020000:01000000:3.0:1788201570.587672:0:42122:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff8e82879e2000: Adding target lustre-OST0000_UUID 00020000:01000000:3.0:1788201570.587677:0:42122:0:(lov_obd.c:532:lov_add_target()) idx=0 ltd_gen=1 ld_tgt_count=1 00000020:01000000:3.0:1788201570.587686:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788201570.587689:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201570.587690:0:42122:0:(obd_config.c:1563:class_process_config()) marker 24 (0x2) lustre-OST0000 add osc 00000020:01000000:3.0:1788201570.587693:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x5 00000020:01000000:3.0:1788201570.587695:0:42122:0:(obd_config.c:1902:class_config_llog_handler()) SKIP #28 00000020:01000000:3.0:1788201570.587696:0:42122:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:3.0:1788201570.587698:0:42122:0:(obd_config.c:1928:class_config_llog_handler()) skipping 0x6 00000020:01000000:3.0:1788201570.587700:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x6 mark_flg=0x6 00000020:00000080:3.0:1788201570.587702:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201570.587703:0:42122:0:(obd_config.c:1563:class_process_config()) marker 28 (0x6) lustre sys.timeout 00000020:01000000:3.0:1788201570.587706:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:01000004:3.0:1788201570.587709:0:42122:0:(obd_mount.c:961:lustre_check_exclusion()) Check exclusion lustre-OST0001 (1) in 0 of 192.168.201.121@tcp:/lustre 00000020:00000080:3.0:1788201570.587711:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201570.587713:0:42122:0:(obd_config.c:1563:class_process_config()) marker 32 (0x1) lustre-OST0001 add osc 00000020:00000080:3.0:1788201570.587716:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf005 00000020:00000080:3.0:1788201570.587718:0:42122:0:(obd_config.c:1487:class_process_config()) adding mapping from uuid 192.168.201.121@tcp to nid 0x20000c0a8c979 (192.168.201.121@tcp) 00000020:01000000:3.0:1788201570.587730:0:42122:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf001, instance name: lustre-OST0001-osc-ffff8e82879e2000 00000020:00000080:3.0:1788201570.587732:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf001 00000020:00000080:3.0:1788201570.587749:0:42122:0:(genops.c:413:class_newdev()) Allocate new device lustre-OST0001-osc-ffff8e82879e2000 (ffff8e828961b370) 00000020:00000080:3.0:1788201570.587759:0:42122:0:(obd_config.c:720:class_attach_name()) OBD: dev 12 attached type osc with refcount 2 00000020:01000000:3.0:1788201570.587764:0:42122:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf003, instance name: lustre-OST0001-osc-ffff8e82879e2000 00000020:00000080:3.0:1788201570.587767:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf003 00010000:00080000:3.0:1788201570.587827:0:42122:0:(ldlm_lib.c:102:import_set_conn()) imp ffff8e82911cf800@lustre-OST0001-osc-ffff8e82879e2000: add connection 192.168.201.121@tcp at head 00010000:00010000:3.0:1788201570.587982:0:42122:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-lustre-OST0001-osc-ffff8e82879e2000-8 is initialized 00000020:00000080:3.0:1788201570.588163:0:42122:0:(obd_config.c:826:class_setup()) finished setup of obd lustre-OST0001-osc-ffff8e82879e2000 (uuid eaf4444c-5340-4caf-8764-d8e0f2a45321) 00000020:01000000:3.0:1788201570.588170:0:42122:0:(obd_config.c:2004:class_config_llog_handler()) cmd cf00d, instance name: lustre-clilov-ffff8e82879e2000 00000020:00000080:3.0:1788201570.588173:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00d 00020000:01000000:3.0:1788201570.588183:0:42122:0:(lov_obd.c:481:lov_add_target()) uuid:lustre-OST0001_UUID idx:1 gen:1 active:1 00020000:01000000:3.0:1788201570.588188:0:42122:0:(lov_obd.c:510:lov_add_target()) lustre-clilov-ffff8e82879e2000: Adding target lustre-OST0001_UUID 00020000:01000000:3.0:1788201570.588191:0:42122:0:(lov_obd.c:532:lov_add_target()) idx=1 ltd_gen=1 ld_tgt_count=2 00000020:01000000:3.0:1788201570.588198:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788201570.588201:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201570.588203:0:42122:0:(obd_config.c:1563:class_process_config()) marker 32 (0x2) lustre-OST0001 add osc 00000020:01000000:3.0:1788201570.588206:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788201570.588208:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201570.588210:0:42122:0:(obd_config.c:1563:class_process_config()) marker 37 (0x1) lustre sys.timeout 00000020:00000080:3.0:1788201570.588212:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf009 00000020:00000080:3.0:1788201570.588214:0:42122:0:(obd_config.c:1537:class_process_config()) changing lustre timeout from 20 to 20 00000020:01000000:3.0:1788201570.588217:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788201570.588219:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201570.588220:0:42122:0:(obd_config.c:1563:class_process_config()) marker 37 (0x2) lustre sys.timeout 00000100:00100000:3.0:1788201570.588263:0:42122:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289529c00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:42122:1875062187533440:192.168.201.121@tcp:502:llog_process_th.0 00000100:00100000:3.0:1788201570.588386:0:42122:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201570.607567:0:42122:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a5180 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:42122:1875062187533568:192.168.201.121@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788201570.607613:0:42122:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788201570.607623:0:42122:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788201570.607625:0:42122:0:(obd_config.c:1563:class_process_config()) marker 42 (0x2) lustre sys.jobid_var 00000040:00080000:3.0:1788201570.607631:0:42122:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 52 count 51 00000020:01000000:3.0:1788201570.610388:0:42118:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 1-51 (rc=0) 10000000:01000000:3.0:1788201570.610423:0:42118:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.121@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:3.0:1788201570.610447:0:42118:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1a00/0x9ae3109cee9e206d lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc238e expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201570.610459:0:42118:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1a00/0x9ae3109cee9e206d lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc238e expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788201570.610468:0:42118:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8e82879e2000 from 1 10000000:01000000:3.0:1788201570.610471:0:42118:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788201570.610520:0:42118:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8289559a00/0x9ae3109cee9e209e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201570.610528:0:42118:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.121@tcp lock: ffff8e8289559a00/0x9ae3109cee9e209e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201570.610538:0:42118:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.121@tcp lock: ffff8e8289559a00/0x9ae3109cee9e209e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201570.610563:0:42118:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952bb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:42118:1875062187533696:192.168.201.121@tcp:101:mount.lustre.0 00000100:00100000:3.0:1788201570.610661:0:42118:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828a7a0240 going to sleep for 11 seconds 00000100:00100000:3.0:1788201570.623859:0:42118:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952bb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:42118:1875062187533824:192.168.201.121@tcp:256:mount.lustre.0 00000100:00100000:3.0:1788201570.623959:0:42118:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828a7a0240 going to sleep for 11 seconds 00000100:00100000:3.0:1788201570.630176:0:42118:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952bb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:42118:1875062187533824:192.168.201.121@tcp:256:mount.lustre.0 10000000:01000000:3.0:1788201570.631150:0:42118:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.121@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:3.0:1788201570.631174:0:42118:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8289559a00/0x9ae3109cee9e209e lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc2395 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201570.631186:0:42118:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.121@tcp lock: ffff8e8289559a00/0x9ae3109cee9e209e lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc2395 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788201570.631193:0:42118:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff8e82879e2000 from 1 10000000:01000000:3.0:1788201570.631196:0:42118:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:3.0:1788201570.631230:0:42118:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8289559c00/0x9ae3109cee9e20a5 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201570.631238:0:42118:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.121@tcp lock: ffff8e8289559c00/0x9ae3109cee9e20a5 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201570.631247:0:42118:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.121@tcp lock: ffff8e8289559c00/0x9ae3109cee9e20a5 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201570.631270:0:42118:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952bb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:42118:1875062187533952:192.168.201.121@tcp:101:mount.lustre.0 00000100:00100000:3.0:1788201570.631825:0:42118:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828a7a0240 going to sleep for 11 seconds 00000100:00100000:3.0:1788201570.634385:0:42118:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952bb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:42118:1875062187533952:192.168.201.121@tcp:101:mount.lustre.0 00010000:00010000:3.0:1788201570.634411:0:42118:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.121@tcp lock: ffff8e8289559c00/0x9ae3109cee9e20a5 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc239c expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201570.634424:0:42118:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.121@tcp lock: ffff8e8289559c00/0x9ae3109cee9e20a5 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x7cf2114380cc239c expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201570.634617:0:42118:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952bb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:42118:1875062187534080:192.168.201.121@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788201570.634693:0:42118:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828a7a0240 going to sleep for 11 seconds 00000100:00100000:3.0:1788201570.640159:0:42118:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952bb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:42118:1875062187534080:192.168.201.121@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788201570.640233:0:42118:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828952bb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:42118:1875062187534208:192.168.201.121@tcp:503:mount.lustre.0 00000100:00100000:3.0:1788201570.640367:0:42118:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828a7a0240 going to sleep for 11 seconds 00000100:00100000:3.0:1788201570.653774:0:42118:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952bb80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:42118:1875062187534208:192.168.201.121@tcp:503:mount.lustre.0 00000100:00100000:1.0:1788201570.654588:0:42129:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510e00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:42129:1875062187534336:192.168.201.121@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788201570.654868:0:42129:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f50c0 going to sleep for 11 seconds 00000020:01000000:3.0:1788201570.670850:0:42118:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 1-6 (rc=0) 10000000:01000000:3.0:1788201570.670882:0:42118:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.121@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:3.0:1788201570.670915:0:42118:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8289559c00/0x9ae3109cee9e20a5 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc239c expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201570.670928:0:42118:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.121@tcp lock: ffff8e8289559c00/0x9ae3109cee9e20a5 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc239c expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:01000000:3.0:1788201570.670953:0:42118:0:(llite_lib.c:1614:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000020:00000080:3.0:1788201570.670984:0:42118:0:(genops.c:1385:class_connect()) connect: client eaf4444c-5340-4caf-8764-d8e0f2a45321, cookie 0x9ae3109cee9e20ac 00800000:01000000:3.0:1788201570.671008:0:42118:0:(lmv_obd.c:515:lmv_check_connect()) Time to connect eaf4444c-5340-4caf-8764-d8e0f2a45321 to lustre-clilmv-ffff8e82879e2000 00800000:01000000:3.0:1788201570.671018:0:42118:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0000-mdc-ffff8e82879e2000(eaf4444c-5340-4caf-8764-d8e0f2a45321) - lustre-MDT0000_UUID, eaf4444c-5340-4caf-8764-d8e0f2a45321 00000020:00000080:3.0:1788201570.671027:0:42118:0:(genops.c:1385:class_connect()) connect: client eaf4444c-5340-4caf-8764-d8e0f2a45321, cookie 0x9ae3109cee9e20b3 00000100:00080000:3.0:1788201570.671039:0:42118:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e82879e2000: connect to NID 192.168.201.121@tcp last attempt 0 00000100:00080000:3.0:1788201570.671051:0:42118:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e82879e2000: import ffff8e82911ce000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201570.671065:0:42118:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 134217728, lost: 0. 00000100:00100000:3.0:1788201570.671092:0:42118:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828952bb80 x1875062187534592/t0(0) o38->lustre-MDT0000-mdc-ffff8e82879e2000@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201570.671122:0:42118:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import eaf4444c-5340-4caf-8764-d8e0f2a45321->lustre-MDT0000_UUID 00800000:01000000:3.0:1788201570.671227:0:42118:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0000-mdc-ffff8e82879e2000(eaf4444c-5340-4caf-8764-d8e0f2a45321) successfully (5) 00800000:00000004:3.0:1788201570.671231:0:42118:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0000-mdc-ffff8e82879e2000: age 2030, max_age 0 00800000:01000000:3.0:1788201570.671264:0:42118:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0001-mdc-ffff8e82879e2000(eaf4444c-5340-4caf-8764-d8e0f2a45321) - lustre-MDT0001_UUID, eaf4444c-5340-4caf-8764-d8e0f2a45321 00000020:00000080:3.0:1788201570.671273:0:42118:0:(genops.c:1385:class_connect()) connect: client eaf4444c-5340-4caf-8764-d8e0f2a45321, cookie 0x9ae3109cee9e20ba 00000100:00080000:3.0:1788201570.671278:0:42118:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e82879e2000: connect to NID 192.168.201.121@tcp last attempt 0 00000100:00080000:3.0:1788201570.671282:0:42118:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e82879e2000: import ffff8e82911c8000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201570.671287:0:42118:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 134217728, lost: 0. 00000100:00100000:3.0:1788201570.671297:0:42118:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289553b80 x1875062187534848/t0(0) o38->lustre-MDT0001-mdc-ffff8e82879e2000@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201570.671329:0:42118:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import eaf4444c-5340-4caf-8764-d8e0f2a45321->lustre-MDT0001_UUID 00800000:01000000:3.0:1788201570.671362:0:42118:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0001-mdc-ffff8e82879e2000(eaf4444c-5340-4caf-8764-d8e0f2a45321) successfully (5) 00800000:00000004:3.0:1788201570.671365:0:42118:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0001-mdc-ffff8e82879e2000: age 2030, max_age 0 00000080:00000004:3.0:1788201570.671379:0:42118:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff8e82879e2000: age 2030, max_age 3029 00800000:00000004:3.0:1788201570.671384:0:42118:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8e82879e2000: age 2030, max_age 3029 00000100:00080000:3.0:1788201570.671397:0:42118:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8e8289550e00 x1875062187535104/t0(0) o41->lustre-MDT0000-mdc-ffff8e82879e2000@192.168.201.121@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201570.671407:0:42118:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b480 going to sleep for 0 seconds 00000100:00080000:3.0:1788201570.671460:0:2403:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8e8289551180 x1875062187534976/t0(0) o41->lustre-MDT0001-mdc-ffff8e82879e2000@192.168.201.121@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201570.671470:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d715180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187535360:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201570.677189:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d716300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187535616:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201570.677303:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00080000:3.0:1788201570.677326:0:2403:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8e8289529c00 x1875062187534720/t0(0) o41->lustre-MDT0000-mdc-ffff8e82879e2000@192.168.201.121@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201570.677339:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289552680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187535232:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201570.677362:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d717800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187535488:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201570.684943:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d717800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187535488:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201570.684961:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187535616:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201570.690009:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e82879e2000: committing for last_committed 0 gen 1 00000100:00080000:1.0:1788201570.690039:0:2400:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8e82879e2000: connect to target with instance 19 00000100:00080000:1.0:1788201570.690050:0:2400:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-MDT0000-mdc-ffff8e82879e2000: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:1.0:1788201570.690064:0:2400:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0000_UUID 00000100:00080000:1.0:1788201570.690068:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82911ce000 lustre-MDT0000_UUID: changing import state from CONNECTING to FULL 00000100:00080000:0.0:1788201570.690341:0:2400:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000002d741750): req@ffff8e8289550e00 x1875062187535104/t0(0) o41->lustre-MDT0000-mdc-ffff8e82879e2000@192.168.201.121@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201570.690368:0:2400:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000004f7d089e): req@ffff8e8289529c00 x1875062187534720/t0(0) o41->lustre-MDT0000-mdc-ffff8e82879e2000@192.168.201.121@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201570.690377:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828952bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eaf4444c-5340-4caf-8764-d8e0f2a45321:2400:1875062187534592:192.168.201.121@tcp:38:mount.lustre.0 00000100:00100000:0.0:1788201570.690500:0:42118:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201570.693839:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8e82879e2000: committing for last_committed 0 gen 1 00000100:00080000:1.0:1788201570.693854:0:2400:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0001-mdc-ffff8e82879e2000: connect to target with instance 1 00000100:00080000:1.0:1788201570.693862:0:2400:0:(import.c:956:ptlrpc_connect_set_flags()) lustre-MDT0001-mdc-ffff8e82879e2000: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:1.0:1788201570.693874:0:2400:0:(import.c:1213:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0001_UUID 00000100:00080000:1.0:1788201570.693878:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82911c8000 lustre-MDT0001_UUID: changing import state from CONNECTING to FULL 00000100:00100000:1.0:1788201570.694216:0:42118:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289550e00 pname:cluuid:pid:xid:nid:opc:job mount.lustre:eaf4444c-5340-4caf-8764-d8e0f2a45321:42118:1875062187535104:192.168.201.121@tcp:41:mount.lustre.0 00800000:00000004:1.0:1788201570.694241:0:42118:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8e82879e2000: update ffff8e82856f8450 cache blocks 317373/353529 objects 1023589/1024000 00000080:00000004:1.0:1788201570.694248:0:42118:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff8e82879e2000: update ffff8e82856fd178 cache blocks 317373/353529 objects 1023589/1024000 00000100:00100000:1.0:1788201570.694282:0:42118:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512680 pname:cluuid:pid:xid:nid:opc:job mount.lustre:eaf4444c-5340-4caf-8764-d8e0f2a45321:42118:1875062187535744:192.168.201.121@tcp:40:mount.lustre.0 00000100:00100000:1.0:1788201570.694369:0:42118:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f53c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201570.715943:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289529c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:eaf4444c-5340-4caf-8764-d8e0f2a45321:0:1875062187534720:192.168.201.121@tcp:41:mount.lustre.0 00000100:00100000:3.0:1788201570.725711:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289551180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:eaf4444c-5340-4caf-8764-d8e0f2a45321:0:1875062187534976:192.168.201.121@tcp:41:mount.lustre.0 00000100:00100000:0.0:1788201570.729470:0:42118:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512680 pname:cluuid:pid:xid:nid:opc:job mount.lustre:eaf4444c-5340-4caf-8764-d8e0f2a45321:42118:1875062187535744:192.168.201.121@tcp:40:mount.lustre.0 00000080:00000004:0.0:1788201570.729564:0:42118:0:(llite_lib.c:614:client_common_fill_super()) rootfid [0x200000007:0x1:0x0] 00000080:00100000:0.0:1788201570.729580:0:42118:0:(llite_lib.c:775:client_common_fill_super()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 2146304 00020000:01000000:0.0:1788201570.729584:0:42118:0:(lov_obd.c:186:lov_connect()) connect #0 00000020:00000080:0.0:1788201570.729723:0:42118:0:(genops.c:1385:class_connect()) connect: client eaf4444c-5340-4caf-8764-d8e0f2a45321, cookie 0x9ae3109cee9e20c1 00000020:00000080:0.0:1788201570.729747:0:42118:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x9ae3109cee9e20c8 00000100:00080000:0.0:1788201570.729754:0:42118:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e82879e2000: connect to NID 192.168.201.121@tcp last attempt 0 00000100:00080000:0.0:1788201570.729765:0:42118:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e82879e2000: import ffff8e82911c9000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201570.729779:0:42118:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788201570.729808:0:42118:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187535872/t0(0) o8->lustre-OST0000-osc-ffff8e82879e2000@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201570.730171:0:42118:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import eaf4444c-5340-4caf-8764-d8e0f2a45321->lustre-OST0000_UUID 00020000:01000000:0.0:1788201570.730238:0:42118:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 0 lustre-OST0000_UUID (lustre-OST0000-osc-ffff8e82879e2000) active 00000020:00000080:0.0:1788201570.730271:0:42118:0:(genops.c:1385:class_connect()) connect: client LOV_OSC_UUID, cookie 0x9ae3109cee9e20cf 00000100:00080000:0.0:1788201570.730279:0:42118:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e82879e2000: connect to NID 192.168.201.121@tcp last attempt 0 00000100:00080000:0.0:1788201570.730286:0:42118:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e82879e2000: import ffff8e82911cf800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201570.730294:0:42118:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 134217728, lost: 0. 00000100:00100000:0.0:1788201570.730343:0:42118:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d180 x1875062187536000/t0(0) o8->lustre-OST0001-osc-ffff8e82879e2000@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201570.730366:0:42118:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import eaf4444c-5340-4caf-8764-d8e0f2a45321->lustre-OST0001_UUID 00020000:01000000:0.0:1788201570.733544:0:42118:0:(lov_obd.c:159:lov_connect_osc()) Connected tgt idx 1 lustre-OST0001_UUID (lustre-OST0001-osc-ffff8e82879e2000) active 00000100:00100000:0.0:1788201570.733915:0:42118:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job mount.lustre:eaf4444c-5340-4caf-8764-d8e0f2a45321:42118:1875062187536128:192.168.201.121@tcp:900:mount.lustre.0 00000100:00100000:0.0:1788201570.734023:0:42118:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287a74780 going to sleep for 11 seconds 00000100:00100000:0.0:1788201570.753162:0:42118:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job mount.lustre:eaf4444c-5340-4caf-8764-d8e0f2a45321:42118:1875062187536128:192.168.201.121@tcp:900:mount.lustre.0 00000100:00100000:0.0:1788201570.753231:0:42118:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job mount.lustre:eaf4444c-5340-4caf-8764-d8e0f2a45321:42118:1875062187536256:192.168.201.121@tcp:33:mount.lustre.0 00000100:00100000:0.0:1788201570.757651:0:42118:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287a74780 going to sleep for 11 seconds 00000100:00100000:0.0:1788201570.760456:0:42118:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job mount.lustre:eaf4444c-5340-4caf-8764-d8e0f2a45321:42118:1875062187536256:192.168.201.121@tcp:33:mount.lustre.0 00000080:00200000:0.0:1788201570.760589:0:42118:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8) 00000080:00200000:0.0:1788201570.760906:0:42118:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:02000400:0.0:1788201570.761158:0:42118:0:(llite_lib.c:1653:ll_fill_super()) Mounted lustre-client - version 2.17.57_103_g9c33a22 00000080:00000004:0.0:1788201570.761163:0:42118:0:(super25.c:182:lustre_fill_super()) 192.168.201.121@tcp:/lustre: Mount complete 00010000:00010000:0.0:1788201571.526272:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788201572.062187:0:42157:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8e8292531988), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788201572.062196:0:42157:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201572.062225:0:42157:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201572.062238:0:42157:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201572.062250:0:42157:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201572.062258:0:42157:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201572.062263:0:42157:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201572.062266:0:42157:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e205f 00000080:00010000:3.0:1788201572.062273:0:42157:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e205f from it ffffaa1801f73cb8 00010000:00010000:3.0:1788201572.062277:0:42157:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201572.062285:0:42157:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1788201572.062444:0:42157:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8),name=/ 00000002:00010000:3.0:1788201572.062459:0:42157:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201572.062469:0:42157:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289560600/0x9ae3109cee9e20d6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc23c6 expref: -99 pid: 42157 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788201572.062478:0:42157:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289560600/0x9ae3109cee9e20d6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc23c6 expref: -99 pid: 42157 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:3.0:1788201572.062485:0:42157:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201572.062489:0:42157:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c080c8) 00000080:00010000:3.0:1788201572.062492:0:42157:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c080c8) for lock 0x9ae3109cee9e20d6 00000080:00010000:3.0:1788201572.062496:0:42157:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e20d6 from it ffffaa1801f73bb0 00010000:00010000:3.0:1788201572.062499:0:42157:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289560600/0x9ae3109cee9e20d6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc23c6 expref: -99 pid: 42157 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788201572.062506:0:42157:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289560600/0x9ae3109cee9e20d6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc23c6 expref: -99 pid: 42157 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:3.0:1788201572.062515:0:42157:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8), inode mode 41ed mask 44 00000080:00200008:3.0:1788201572.062525:0:42157:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8e8289c080c8), flags 300000 00000080:00200080:3.0:1788201572.062542:0:42157:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8) cmd=c03066f0 arg=7ffe05a72590 00000080:00200000:3.0:1788201572.062554:0:42157:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8) name=d12a.replay-vbr stripe_offset=0 stripe_count=1, hash_type=0 00000100:00100000:3.0:1788201572.062634:0:42157:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d716300 pname:cluuid:pid:xid:nid:opc:job lfs:eaf4444c-5340-4caf-8764-d8e0f2a45321:42157:1875062187536640:192.168.201.121@tcp:700:lfs.0 00000100:00100000:3.0:1788201572.063019:0:42157:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2be40 going to sleep for 11 seconds 00000100:00100000:3.0:1788201572.088047:0:42157:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716300 pname:cluuid:pid:xid:nid:opc:job lfs:eaf4444c-5340-4caf-8764-d8e0f2a45321:42157:1875062187536640:192.168.201.121@tcp:700:lfs.0 00010000:00010000:3.0:1788201572.088101:0:42157:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289560600/0x9ae3109cee9e20d6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc23c6 expref: -99 pid: 42157 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788201572.088168:0:42157:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289560600/0x9ae3109cee9e20d6 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc23c6 expref: -99 pid: 42157 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788201572.088176:0:42157:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201572.088179:0:42157:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289560600/0x9ae3109cee9e20d6 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc23c6 expref: -99 pid: 42157 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000020:00100000:3.0:1788201572.088203:0:42157:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e82879e2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201572.088211:0:42157:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d716300 x1875062187536896/t0(0) o36->lustre-MDT0000-mdc-ffff8e82879e2000@192.168.201.121@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201572.088232:0:42157:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d716300 pname:cluuid:pid:xid:nid:opc:job lfs:eaf4444c-5340-4caf-8764-d8e0f2a45321:42157:1875062187536896:192.168.201.121@tcp:36:lfs.0 00000100:00100000:3.0:1788201572.088353:0:42157:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2be40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201572.096513:0:2393:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.121@tcp (source: 12345-192.168.201.121@tcp) 00000100:00100000:0.0:1788201572.096824:0:6163:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875062212038912 00000100:00100000:0.0:1788201572.096834:0:6163:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875062212038912 00000100:00100000:0.0:1788201572.096902:0:6163:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.121@tcp, seq: 93 00000100:00100000:0.0:1788201572.096913:0:6163:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8e8289511180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:dc31b02e-cf70-4b83-9f70-d7124af83e4d+5:6521:x1875062212038912:12345-192.168.201.121@tcp:104: 00010000:00010000:0.0:1788201572.096930:0:6163:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788201572.097038:0:6163:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:0.0:1788201572.097045:0:6163:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8e8289511180 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:dc31b02e-cf70-4b83-9f70-d7124af83e4d+5:6521:x1875062212038912:12345-192.168.201.121@tcp:104: Request processed in 133us (542us total) trans 0 rc 0/0 00000100:00100000:0.0:1788201572.097053:0:6163:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.121@tcp, seq: 93 00010000:00010000:3.0:1788201572.113845:0:6173:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201572.113859:0:6173:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788201572.113872:0:6173:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8e8290ee0c00 already unused, calling callback (ffffffffc1c51460) 00010000:00010000:3.0:1788201572.113877:0:6173:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788201572.113891:0:6173:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788201572.113900:0:6173:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788201572.114112:0:6173:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 2/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788201572.114121:0:6173:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201572.114157:0:6173:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:0.0:1788201572.114165:0:6173:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0c00/0x9ae3109cee9e205f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x13 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x7cf2114380cc20b6 expref: -99 pid: 40852 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000100:00100000:0.0:1788201572.114187:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788201572.114207:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187537024:192.168.201.121@tcp:103:ldlm_bl.0 00010000:00010000:0.0:1788201572.114451:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201572.114454:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201572.114455:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201572.115959:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187537024:192.168.201.121@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788201572.137476:0:42157:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716300 pname:cluuid:pid:xid:nid:opc:job lfs:eaf4444c-5340-4caf-8764-d8e0f2a45321:42157:1875062187536896:192.168.201.121@tcp:36:lfs.0 00000080:00200000:3.0:1788201572.137528:0:42157:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000090a1:0x1:0x0](ffff8e8289e580c8) 00000080:00200000:3.0:1788201572.137535:0:42157:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000090a1:0x1:0x0], updating i_size 4096 00000080:00200008:3.0:1788201572.137568:0:42157:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8e8289c080c8), flags 300000 00010000:00010000:3.0:1788201572.548964:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788201573.429808:0:42166:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201573.429838:0:42166:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e20e4 00000080:00010000:3.0:1788201573.429846:0:42166:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201573.429850:0:42166:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e20e4 from it ffffaa1801fa3cb8 00010000:00010000:3.0:1788201573.429857:0:42166:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201573.429872:0:42166:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788201573.431274:0:42166:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8e8292531988), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788201573.431286:0:42166:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201573.431335:0:42166:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201573.431353:0:42166:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201573.431373:0:42166:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201573.431382:0:42166:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201573.431390:0:42166:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201573.431394:0:42166:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e20e4 00000080:00010000:3.0:1788201573.431400:0:42166:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e20e4 from it ffffaa1801fa3cb8 00010000:00010000:3.0:1788201573.431404:0:42166:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201573.431414:0:42166:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201573.431549:0:42166:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201573.431558:0:42166:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201573.431565:0:42166:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201573.431573:0:42166:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201573.431579:0:42166:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201573.431595:0:42166:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201573.431598:0:42166:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e20e4 00000080:00010000:3.0:1788201573.431601:0:42166:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e20e4 from it ffffaa1801fa3bb0 00010000:00010000:3.0:1788201573.431604:0:42166:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201573.433387:0:42166:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201573.433407:0:42166:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 44 00000080:00200008:3.0:1788201573.433419:0:42166:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8e8292531988), flags 100000 00000080:00200080:3.0:1788201573.433559:0:42166:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe8e6fc790 00800000:00000080:3.0:1788201573.433576:0:42166:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8289622800 uarg=0000000000000000 00800000:00000004:3.0:1788201573.433594:0:42166:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8e8282a4b800: age 3021, max_age 3032 00000100:00100000:3.0:1788201573.436478:0:42166:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d717800 pname:cluuid:pid:xid:nid:opc:job lfs:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42166:1875062187537280:192.168.201.121@tcp:41:lfs.0 00000100:00100000:3.0:1788201573.436580:0:42166:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2be40 going to sleep for 11 seconds 00000100:00100000:3.0:1788201573.450602:0:42166:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d717800 pname:cluuid:pid:xid:nid:opc:job lfs:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42166:1875062187537280:192.168.201.121@tcp:41:lfs.0 00800000:00000004:3.0:1788201573.450642:0:42166:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8e8282a4b800: update ffff8e8291bdb7c0 cache blocks 317372/353529 objects 1023588/1024000 00000080:00200080:3.0:1788201573.450729:0:42166:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe8e6fc790 00800000:00000080:3.0:1788201573.450747:0:42166:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8289622800 uarg=0000000000000000 00800000:00000004:3.0:1788201573.450755:0:42166:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8e8282a4b800: age 3021, max_age 3032 00000100:00100000:3.0:1788201573.450789:0:42166:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d717800 pname:cluuid:pid:xid:nid:opc:job lfs:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42166:1875062187537408:192.168.201.121@tcp:41:lfs.0 00000080:00200000:0.0:1788201573.575418:0:42173:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201573.575447:0:42173:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201573.575465:0:42173:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201573.575481:0:42173:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201573.575489:0:42173:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201573.575496:0:42173:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201573.575500:0:42173:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e20e4 00000080:00010000:0.0:1788201573.575507:0:42173:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e20e4 from it ffffaa1801fb3bd0 00010000:00010000:0.0:1788201573.575511:0:42173:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201573.575521:0:42173:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:0.0:1788201573.575534:0:42173:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201573.575546:0:42173:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg121-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=1536 00000080:00200000:0.0:1788201573.575550:0:42173:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201573.575556:0:42173:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201573.575563:0:42173:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788201573.575572:0:42173:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:0.0:1788201573.575578:0:42173:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201573.575601:0:42173:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201573.575604:0:42173:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e20e4 00000080:00010000:0.0:1788201573.575608:0:42173:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e20e4 from it ffffaa1801fb3cd0 00010000:00010000:0.0:1788201573.575611:0:42173:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:0.0:1788201573.575618:0:42173:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:0.0:1788201573.575626:0:42173:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201573.575631:0:42173:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201573.575638:0:42173:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201573.575644:0:42173:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788201573.575652:0:42173:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:0.0:1788201573.575658:0:42173:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201573.575661:0:42173:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201573.575663:0:42173:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e20e4 00000080:00010000:0.0:1788201573.575666:0:42173:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e20e4 from it ffffaa1801fb3db0 00010000:00010000:0.0:1788201573.575669:0:42173:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:0.0:1788201573.575676:0:42173:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:0.0:1788201573.575682:0:42173:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201573.575687:0:42173:0:(namei.c:2129:ll_create_nd()) VFS Op:name=fsa-oleg121-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=33188, excl=1 00000080:00200000:0.0:1788201573.575690:0:42173:0:(namei.c:2084:ll_mknod()) VFS Op:name=fsa-oleg121-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8e8292531988) mode 100644 dev 0 00010000:00010000:0.0:1788201573.575716:0:42173:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:0.0:1788201573.575773:0:42173:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:0.0:1788201573.575781:0:42173:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201573.575784:0:42173:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e20e4 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc23fe expref: -99 pid: 42166 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000020:00100000:0.0:1788201573.575813:0:42173:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201573.575827:0:42173:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793c700 x1875062187537920/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201573.575851:0:42173:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job mcreate:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42173:1875062187537920:192.168.201.121@tcp:36:mcreate.0 00000100:00100000:0.0:1788201573.575947:0:42173:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287a749c0 going to sleep for 11 seconds 00010000:00010000:0.0:1788201573.580064:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201573.582932:0:42173:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job mcreate:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42173:1875062187537920:192.168.201.121@tcp:36:mcreate.0 00000080:00200000:3.0:1788201573.582983:0:42173:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xd7:0x0](ffff8e8289e5ba88) 00000080:00200000:3.0:1788201573.583032:0:42173:0:(namei.c:2137:ll_create_nd()) VFS Op:name=fsa-oleg121-client.virtnet, unhashed 1 00000080:00200000:1.0:1788201573.599571:0:42176:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201573.599616:0:42176:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201573.599676:0:42176:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42176 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201573.599686:0:42176:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42176 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201573.599695:0:42176:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42176 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201573.599717:0:42176:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job rm:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42176:1875062187538048:192.168.201.121@tcp:101:rm.0 00000100:00100000:1.0:1788201573.599863:0:42176:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f5600 going to sleep for 11 seconds 00000100:00100000:3.0:1788201573.606376:0:42176:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job rm:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42176:1875062187538048:192.168.201.121@tcp:101:rm.0 00010000:00010000:3.0:1788201573.606404:0:42176:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201573.606477:0:42176:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201573.606551:0:42176:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201573.606561:0:42176:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201573.606574:0:42176:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201573.606592:0:42176:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201573.606599:0:42176:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201573.606610:0:42176:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289511c00 x1875062187538048/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201589 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201573.606623:0:42176:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201573.606634:0:42176:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201573.606640:0:42176:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201573.606667:0:42176:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201573.606673:0:42176:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e20eb 00000080:00010000:3.0:1788201573.606679:0:42176:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201573.606682:0:42176:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e20eb from it ffffaa1801fb3b58 00010000:00010000:3.0:1788201573.606685:0:42176:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201573.606693:0:42176:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201573.606717:0:42176:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201573.606737:0:42176:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg121-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=8 00000080:00200000:3.0:1788201573.606741:0:42176:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg121-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201573.606761:0:42176:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: fsa-oleg121-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201573.606799:0:42176:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42176 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201573.606806:0:42176:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42176 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201573.606815:0:42176:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42176 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201573.606835:0:42176:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d717480 pname:cluuid:pid:xid:nid:opc:job rm:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42176:1875062187538176:192.168.201.121@tcp:101:rm.0 00000100:00100000:3.0:1788201573.607110:0:42176:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b780 going to sleep for 11 seconds 00000100:00100000:3.0:1788201573.611806:0:42176:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d717480 pname:cluuid:pid:xid:nid:opc:job rm:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42176:1875062187538176:192.168.201.121@tcp:101:rm.0 00010000:00010000:3.0:1788201573.611834:0:42176:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2436 expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201573.611863:0:42176:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xd7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2436 expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201573.611872:0:42176:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xd7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2436 expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201573.611884:0:42176:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xd7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2436 expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201573.611895:0:42176:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xd7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2436 expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201573.611906:0:42176:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xd7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2436 expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201573.611915:0:42176:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xd7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2436 expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201573.611922:0:42176:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xd7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2436 expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201573.611933:0:42176:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d717480 x1875062187538176/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 600/584 e 0 to 0 dl 1788201589 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201573.611944:0:42176:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xd7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2436 expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201573.611954:0:42176:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/215 (0 0) 00000080:00200000:3.0:1788201573.611961:0:42176:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201573.611983:0:42176:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200006991:0xd7:0x0], updating i_size 0 00000080:00200000:3.0:1788201573.611986:0:42176:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xd7:0x0](ffff8e8289e5ba88): rc = 0 00000080:00200000:3.0:1788201573.612142:0:42176:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xd7:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1788201573.612149:0:42176:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xd7:0x0](ffff8e8289e5ba88) for lock 0x9ae3109cee9e20f2 00000080:00010000:3.0:1788201573.612160:0:42176:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xd7:0x0](ffff8e8289e5ba88) 00000080:00010000:3.0:1788201573.612163:0:42176:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e20f2 from it ffffaa1801fb3b20 00010000:00010000:3.0:1788201573.612166:0:42176:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xd7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc2436 expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201573.612176:0:42176:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xd7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc2436 expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788201573.612212:0:42176:0:(file.c:6296:ll_getattr_dentry()) START file fsa-oleg121-client.virtnet:[0x200006991:0xd7:0x0](ffff8e8289e5ba88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201573.612219:0:42176:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200006991:0xd7:0x0](ffff8e8289e5ba88),name=fsa-oleg121-client.virtnet 00000002:00010000:3.0:1788201573.612243:0:42176:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200006991:0xd7:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201573.612254:0:42176:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 2/1,0 mode: PR/PR res: [0x200006991:0xd7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc2436 expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201573.612265:0:42176:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 2/1,0 mode: PR/PR res: [0x200006991:0xd7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc2436 expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201573.612271:0:42176:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201573.612276:0:42176:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xd7:0x0](ffff8e8289e5ba88) 00000080:00010000:3.0:1788201573.612278:0:42176:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xd7:0x0](ffff8e8289e5ba88) for lock 0x9ae3109cee9e20f2 00000080:00010000:3.0:1788201573.612282:0:42176:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e20f2 from it ffffaa1801fb3cb8 00010000:00010000:3.0:1788201573.612285:0:42176:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xd7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc2436 expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201573.612292:0:42176:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xd7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc2436 expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201573.612473:0:42176:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201573.612549:0:42176:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201573.612557:0:42176:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201573.612566:0:42176:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201573.612573:0:42176:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201573.612578:0:42176:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201573.612589:0:42176:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e20eb 00000080:00010000:3.0:1788201573.612593:0:42176:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e20eb from it ffffaa1801fb3c10 00010000:00010000:3.0:1788201573.612596:0:42176:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201573.612603:0:42176:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201573.612611:0:42176:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201573.612620:0:42176:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg121-client.virtnet, flags=0 00000080:00200000:3.0:1788201573.612624:0:42176:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201573.612631:0:42176:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201573.612637:0:42176:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201573.612645:0:42176:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201573.612651:0:42176:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201573.612655:0:42176:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201573.612657:0:42176:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e20eb 00000080:00010000:3.0:1788201573.612660:0:42176:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e20eb from it ffffaa1801fb3d68 00010000:00010000:3.0:1788201573.612663:0:42176:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201573.612669:0:42176:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788201573.612677:0:42176:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201573.612682:0:42176:0:(namei.c:2418:ll_unlink()) VFS Op:name=fsa-oleg121-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201573.612702:0:42176:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201573.612725:0:42176:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xd7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc2436 expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201573.612734:0:42176:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xd7:0x0]: losing layout lock 00010000:00010000:3.0:1788201573.612765:0:42176:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xd7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc2436 expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201573.612772:0:42176:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788201573.612782:0:42176:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201573.612785:0:42176:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e20f2 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xd7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc2436 expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201573.612799:0:42176:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993a00/0x9ae3109cee9e20eb lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc241a expref: -99 pid: 42176 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:3.0:1788201573.612817:0:42176:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201573.612825:0:42176:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d717480 x1875062187538432/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201573.612847:0:42176:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d717480 pname:cluuid:pid:xid:nid:opc:job rm:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42176:1875062187538432:192.168.201.121@tcp:36:rm.0 00000100:00100000:3.0:1788201573.612928:0:42176:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b780 going to sleep for 11 seconds 00000100:00100000:3.0:1788201573.616888:0:42176:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d717480 pname:cluuid:pid:xid:nid:opc:job rm:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42176:1875062187538432:192.168.201.121@tcp:36:rm.0 00000080:00200000:3.0:1788201573.616942:0:42176:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xd7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788201573.616952:0:42176:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xd7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201573.616960:0:42176:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xd7:0x0](ffff8e8289e5ba88) 00000080:00200008:1.0:1788201573.654230:0:42180:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8e8289c080c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201573.654245:0:42180:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8),name=/ 00000002:00010000:1.0:1788201573.654272:0:42180:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00000080:00200000:3.0:1788201573.676098:0:42183:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8),name=/ 00000002:00010000:3.0:1788201573.676132:0:42183:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201573.676149:0:42183:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289561200/0x9ae3109cee9e20f9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2460 expref: -99 pid: 42180 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201573.676166:0:42183:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289561200/0x9ae3109cee9e20f9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2460 expref: -99 pid: 42180 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201573.676175:0:42183:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201573.676184:0:42183:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c080c8) 00000080:00010000:3.0:1788201573.676188:0:42183:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c080c8) for lock 0x9ae3109cee9e20f9 00000080:00010000:3.0:1788201573.676196:0:42183:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e20f9 from it ffffaa1801fc3bd0 00010000:00010000:3.0:1788201573.676200:0:42183:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289561200/0x9ae3109cee9e20f9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2460 expref: -99 pid: 42180 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201573.676209:0:42183:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289561200/0x9ae3109cee9e20f9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2460 expref: -99 pid: 42180 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201573.676225:0:42183:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788201573.676240:0:42183:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg121-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8e8289c080c8), flags=1536 00000080:00200000:3.0:1788201573.676244:0:42183:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8),name=/ 00000002:00010000:3.0:1788201573.676251:0:42183:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201573.676256:0:42183:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289561200/0x9ae3109cee9e20f9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2460 expref: -99 pid: 42180 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201573.676264:0:42183:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289561200/0x9ae3109cee9e20f9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2460 expref: -99 pid: 42180 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201573.676271:0:42183:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201573.676275:0:42183:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c080c8) 00000080:00010000:3.0:1788201573.676277:0:42183:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c080c8) for lock 0x9ae3109cee9e20f9 00000080:00010000:3.0:1788201573.676281:0:42183:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e20f9 from it ffffaa1801fc3cd0 00010000:00010000:3.0:1788201573.676285:0:42183:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289561200/0x9ae3109cee9e20f9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2460 expref: -99 pid: 42180 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201573.676292:0:42183:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289561200/0x9ae3109cee9e20f9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2460 expref: -99 pid: 42180 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1788201573.676304:0:42183:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8), inode mode 41ed mask 3 00000080:00200000:3.0:1788201573.676644:0:42183:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8),name=/ 00000002:00010000:3.0:1788201573.676653:0:42183:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201573.676660:0:42183:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289561200/0x9ae3109cee9e20f9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2460 expref: -99 pid: 42180 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788201573.676667:0:42183:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289561200/0x9ae3109cee9e20f9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2460 expref: -99 pid: 42180 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1788201573.676673:0:42183:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201573.676678:0:42183:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c080c8) 00000080:00010000:3.0:1788201573.676680:0:42183:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c080c8) for lock 0x9ae3109cee9e20f9 00000080:00010000:3.0:1788201573.676684:0:42183:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e20f9 from it ffffaa1801fc3db0 00010000:00010000:3.0:1788201573.676686:0:42183:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289561200/0x9ae3109cee9e20f9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2460 expref: -99 pid: 42180 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788201573.676693:0:42183:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289561200/0x9ae3109cee9e20f9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2460 expref: -99 pid: 42180 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:3.0:1788201573.676701:0:42183:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8), inode mode 41ed mask 3 00000080:00200000:3.0:1788201573.676705:0:42183:0:(namei.c:2129:ll_create_nd()) VFS Op:name=fsa-oleg121-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8e8289c080c8), flags=33188, excl=1 00000080:00200000:3.0:1788201573.676709:0:42183:0:(namei.c:2084:ll_mknod()) VFS Op:name=fsa-oleg121-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8e8289c080c8) mode 100644 dev 0 00010000:00010000:3.0:1788201573.676733:0:42183:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289561200/0x9ae3109cee9e20f9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc2460 expref: -99 pid: 42180 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788201573.676781:0:42183:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289561200/0x9ae3109cee9e20f9 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2460 expref: -99 pid: 42180 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788201573.676788:0:42183:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201573.676790:0:42183:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289561200/0x9ae3109cee9e20f9 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2460 expref: -99 pid: 42180 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000020:00100000:3.0:1788201573.676812:0:42183:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e82879e2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201573.676822:0:42183:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d715500 x1875062187538816/t0(0) o36->lustre-MDT0000-mdc-ffff8e82879e2000@192.168.201.121@tcp:12/10 lens 528/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201573.676845:0:42183:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d715500 pname:cluuid:pid:xid:nid:opc:job mcreate:eaf4444c-5340-4caf-8764-d8e0f2a45321:42183:1875062187538816:192.168.201.121@tcp:36:mcreate.0 00000100:00100000:3.0:1788201573.676924:0:42183:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b780 going to sleep for 11 seconds 00000100:00100000:3.0:1788201573.680756:0:42183:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d715500 pname:cluuid:pid:xid:nid:opc:job mcreate:eaf4444c-5340-4caf-8764-d8e0f2a45321:42183:1875062187538816:192.168.201.121@tcp:36:mcreate.0 00000080:00200000:3.0:1788201573.680804:0:42183:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x2000090a1:0x2:0x0](ffff8e8289e5ba88) 00000080:00200000:3.0:1788201573.680841:0:42183:0:(namei.c:2137:ll_create_nd()) VFS Op:name=fsa-oleg121-client.virtnet, unhashed 1 00000080:00200000:0.0:1788201573.695680:0:42186:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8),name=/ 00000002:00010000:0.0:1788201573.695712:0:42186:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201573.695775:0:42186:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42186 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.695785:0:42186:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42186 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.695795:0:42186:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42186 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201573.695818:0:42186:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job rm:eaf4444c-5340-4caf-8764-d8e0f2a45321:42186:1875062187538944:192.168.201.121@tcp:101:rm.0 00000100:00100000:0.0:1788201573.695900:0:42186:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287a74180 going to sleep for 11 seconds 00000100:00100000:0.0:1788201573.698802:0:42186:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job rm:eaf4444c-5340-4caf-8764-d8e0f2a45321:42186:1875062187538944:192.168.201.121@tcp:101:rm.0 00010000:00010000:0.0:1788201573.698828:0:42186:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.698840:0:42186:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.698857:0:42186:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.698871:0:42186:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.698883:0:42186:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.698892:0:42186:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.698899:0:42186:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201573.698911:0:42186:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793dc00 x1875062187538944/t0(0) o101->lustre-MDT0000-mdc-ffff8e82879e2000@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201589 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201573.698924:0:42186:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.698937:0:42186:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e829232b000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:0.0:1788201573.698944:0:42186:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201573.699189:0:42186:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201573.699197:0:42186:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c080c8) for lock 0x9ae3109cee9e2100 00000080:00010000:0.0:1788201573.699202:0:42186:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c080c8) 00000080:00010000:0.0:1788201573.699206:0:42186:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2100 from it ffffaa1801fd3b58 00010000:00010000:0.0:1788201573.699209:0:42186:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.699218:0:42186:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201573.699242:0:42186:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201573.699262:0:42186:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg121-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8e8289c080c8), flags=8 00000080:00200000:0.0:1788201573.699267:0:42186:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg121-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8e8289c080c8), intent=getattr 00000002:00010000:0.0:1788201573.699288:0:42186:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: fsa-oleg121-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201573.699601:0:42186:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42186 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.699609:0:42186:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42186 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.699618:0:42186:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42186 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201573.699638:0:42186:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job rm:eaf4444c-5340-4caf-8764-d8e0f2a45321:42186:1875062187539072:192.168.201.121@tcp:101:rm.0 00000100:00100000:0.0:1788201573.699751:0:42186:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287a74180 going to sleep for 11 seconds 00000100:00100000:0.0:1788201573.703254:0:42186:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job rm:eaf4444c-5340-4caf-8764-d8e0f2a45321:42186:1875062187539072:192.168.201.121@tcp:101:rm.0 00010000:00010000:0.0:1788201573.703278:0:42186:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2498 expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.703298:0:42186:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 4/1,0 mode: --/PR res: [0x2000090a1:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2498 expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.703551:0:42186:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 4/1,0 mode: --/PR res: [0x2000090a1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2498 expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.703562:0:42186:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 4/1,0 mode: PR/PR res: [0x2000090a1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2498 expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.703573:0:42186:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 4/1,0 mode: PR/PR res: [0x2000090a1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2498 expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.703593:0:42186:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 5/2,0 mode: PR/PR res: [0x2000090a1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2498 expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.703601:0:42186:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 5/2,0 mode: PR/PR res: [0x2000090a1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2498 expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.703609:0:42186:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 4/1,0 mode: PR/PR res: [0x2000090a1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2498 expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201573.703620:0:42186:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793dc00 x1875062187539072/t0(0) o101->lustre-MDT0000-mdc-ffff8e82879e2000@192.168.201.121@tcp:12/10 lens 600/584 e 0 to 0 dl 1788201589 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201573.703633:0:42186:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2498 expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.703643:0:42186:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e829232b000 type 13 mode 4 res 8589971617/2 (0 0) 00000080:00200000:0.0:1788201573.703650:0:42186:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201573.703669:0:42186:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000090a1:0x2:0x0], updating i_size 0 00000080:00200000:0.0:1788201573.703673:0:42186:0:(namei.c:147:ll_iget()) got inode: [0x2000090a1:0x2:0x0](ffff8e8289e5ba88): rc = 0 00000080:00200000:0.0:1788201573.703689:0:42186:0:(file.c:6843:ll_layout_conf()) [0x2000090a1:0x2:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:0.0:1788201573.703694:0:42186:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000090a1:0x2:0x0](ffff8e8289e5ba88) for lock 0x9ae3109cee9e2107 00000080:00010000:0.0:1788201573.703705:0:42186:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000090a1:0x2:0x0](ffff8e8289e5ba88) 00000080:00010000:0.0:1788201573.703708:0:42186:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2107 from it ffffaa1801fd3b20 00010000:00010000:0.0:1788201573.703712:0:42186:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc2498 expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201573.703720:0:42186:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc2498 expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:0.0:1788201573.703755:0:42186:0:(file.c:6296:ll_getattr_dentry()) START file fsa-oleg121-client.virtnet:[0x2000090a1:0x2:0x0](ffff8e8289e5ba88), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788201573.703762:0:42186:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x2000090a1:0x2:0x0](ffff8e8289e5ba88),name=fsa-oleg121-client.virtnet 00000100:00100000:0.0:1788201573.703811:0:42186:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job rm:eaf4444c-5340-4caf-8764-d8e0f2a45321:42186:1875062187539200:192.168.201.121@tcp:900:rm.0 00000100:00100000:0.0:1788201573.703918:0:42186:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287a74180 going to sleep for 11 seconds 00000100:00100000:0.0:1788201573.706186:0:42186:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job rm:eaf4444c-5340-4caf-8764-d8e0f2a45321:42186:1875062187539200:192.168.201.121@tcp:900:rm.0 00000002:00010000:0.0:1788201573.706219:0:42186:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x2000090a1:0x2:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201573.706236:0:42186:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc2498 expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201573.706250:0:42186:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc2498 expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201573.706258:0:42186:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201573.706266:0:42186:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000090a1:0x2:0x0](ffff8e8289e5ba88) 00000080:00010000:0.0:1788201573.706269:0:42186:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000090a1:0x2:0x0](ffff8e8289e5ba88) for lock 0x9ae3109cee9e2107 00000080:00010000:0.0:1788201573.706277:0:42186:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2107 from it ffffaa1801fd3cb8 00010000:00010000:0.0:1788201573.706280:0:42186:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc2498 expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201573.706289:0:42186:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc2498 expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201573.706475:0:42186:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8),name=/ 00000002:00010000:0.0:1788201573.706549:0:42186:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201573.706559:0:42186:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201573.706567:0:42186:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201573.706573:0:42186:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201573.706578:0:42186:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c080c8) 00000080:00010000:0.0:1788201573.706591:0:42186:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c080c8) for lock 0x9ae3109cee9e2100 00000080:00010000:0.0:1788201573.706596:0:42186:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2100 from it ffffaa1801fd3c10 00010000:00010000:0.0:1788201573.706599:0:42186:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201573.706605:0:42186:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201573.706614:0:42186:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8), inode mode 41ed mask 1 00000080:00200000:0.0:1788201573.706623:0:42186:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg121-client.virtnet, flags=0 00000080:00200000:0.0:1788201573.706629:0:42186:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8),name=/ 00000002:00010000:0.0:1788201573.706635:0:42186:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201573.706641:0:42186:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201573.706650:0:42186:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201573.706657:0:42186:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201573.706660:0:42186:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c080c8) 00000080:00010000:0.0:1788201573.706663:0:42186:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c080c8) for lock 0x9ae3109cee9e2100 00000080:00010000:0.0:1788201573.706666:0:42186:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2100 from it ffffaa1801fd3d68 00010000:00010000:0.0:1788201573.706669:0:42186:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201573.706676:0:42186:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788201573.706683:0:42186:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8), inode mode 41ed mask 3 00000080:00200000:0.0:1788201573.706689:0:42186:0:(namei.c:2418:ll_unlink()) VFS Op:name=fsa-oleg121-client.virtnet, dir=[0x200000007:0x1:0x0](ffff8e8289c080c8) 00010000:00010000:0.0:1788201573.706716:0:42186:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201573.706737:0:42186:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc2498 expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201573.706748:0:42186:0:(vvp_object.c:117:vvp_conf_set()) [0x2000090a1:0x2:0x0]: losing layout lock 00010000:00010000:0.0:1788201573.706779:0:42186:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 1/0,0 mode: --/PR res: [0x2000090a1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc2498 expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201573.706786:0:42186:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788201573.706797:0:42186:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201573.706800:0:42186:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1200/0x9ae3109cee9e2107 lrc: 0/0,0 mode: --/PR res: [0x2000090a1:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc2498 expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201573.706815:0:42186:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8290ee1800/0x9ae3109cee9e2100 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc247c expref: -99 pid: 42186 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1788201573.706832:0:42186:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e82879e2000: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201573.706840:0:42186:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793dc00 x1875062187539456/t0(0) o36->lustre-MDT0000-mdc-ffff8e82879e2000@192.168.201.121@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201573.706862:0:42186:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job rm:eaf4444c-5340-4caf-8764-d8e0f2a45321:42186:1875062187539456:192.168.201.121@tcp:36:rm.0 00000100:00100000:0.0:1788201573.706972:0:42186:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287a74180 going to sleep for 11 seconds 00010000:00010000:1.0:1788201574.606716:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201575.621471:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201575.942252:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187539712:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201575.942370:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187539968:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201575.942396:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:eaf4444c-5340-4caf-8764-d8e0f2a45321:2402:1875062187540224:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201575.942416:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:eaf4444c-5340-4caf-8764-d8e0f2a45321:2402:1875062187540480:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201575.942443:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 5 async RPCs [0->1] 00000100:00100000:0.0:1788201575.942449:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187539584:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201575.942473:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187539840:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201575.942495:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187540096:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201575.942515:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:eaf4444c-5340-4caf-8764-d8e0f2a45321:2402:1875062187540352:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201575.942558:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:eaf4444c-5340-4caf-8764-d8e0f2a45321:2402:1875062187540608:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201575.954426:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187539584:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201575.954447:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187539840:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201575.954459:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187540096:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201575.954469:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:eaf4444c-5340-4caf-8764-d8e0f2a45321:2402:1875062187540352:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201575.954483:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:eaf4444c-5340-4caf-8764-d8e0f2a45321:2402:1875062187540608:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201575.954495:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187539712:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201575.954505:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187539968:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201575.954514:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:eaf4444c-5340-4caf-8764-d8e0f2a45321:2402:1875062187540224:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201575.954539:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:eaf4444c-5340-4caf-8764-d8e0f2a45321:2402:1875062187540480:192.168.201.121@tcp:400:kworker.0 00010000:00010000:0.0:1788201576.644798:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201577.668904:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201578.693525:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201579.716831:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788201579.896945:0:42248:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00000080:00200000:1.0:1788201580.018239:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201580.018267:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201580.018576:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e211c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.018588:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e211c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.018598:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e211c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201580.018625:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512680 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187541248:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201580.018751:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f5900 going to sleep for 11 seconds 00000100:00100000:1.0:1788201580.023008:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512680 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187541248:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201580.023034:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e211c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc24fa expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.023047:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e211c lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc24fa expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.023059:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e211c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc24fa expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.023071:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e211c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc24fa expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.023082:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e211c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc24fa expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.023091:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e211c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc24fa expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.023099:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e211c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc24fa expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201580.023111:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289512680 x1875062187541248/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201580.023130:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e211c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc24fa expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.023143:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201580.023149:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201580.023173:0:42256:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201580.023180:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e211c 00000080:00010000:1.0:1788201580.023185:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201580.023189:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e211c from it ffffaa1802033bf8 00010000:00010000:1.0:1788201580.023192:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e211c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc24fa expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.023214:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e211c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc24fa expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201580.023237:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201580.023257:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201580.023270:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201580.023279:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e211c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc24fa expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201580.023290:0:42256:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e211c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc24fa expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201580.023297:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201580.023301:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201580.023343:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e211c 00000080:00010000:1.0:1788201580.023349:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e211c from it ffffaa1802033b38 00010000:00010000:1.0:1788201580.023353:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e211c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc24fa expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201580.023362:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e211c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc24fa expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201580.023371:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201580.023377:0:42256:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-0, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8291da8780, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201580.023385:0:42256:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-0, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:1.0:1788201580.023404:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-0,[0x200006991:0xd8:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201580.023414:0:42256:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e211c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc24fa expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201580.023453:0:42256:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e211c lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc24fa expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201580.023460:0:42256:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201580.023464:0:42256:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e211c lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc24fa expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201580.023509:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e2123 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.023517:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e2123 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201580.023527:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201580.023534:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289512680 x1875062187541504/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201580.023544:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e2123 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201580.023562:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512680 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187541504:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201580.023677:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f5900 going to sleep for 11 seconds 00000100:00100000:0.0:1788201580.026252:0:2394:0:(events.c:340:request_in_callback()) peer: 12345-192.168.201.121@tcp (source: 12345-192.168.201.121@tcp) 00000100:00100000:1.0:1788201580.026403:0:6162:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875062212041600 00000100:00100000:1.0:1788201580.026413:0:6162:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875062212041600 00000100:00100000:1.0:1788201580.026486:0:6162:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.201.121@tcp, seq: 94 00000100:00100000:1.0:1788201580.026497:0:6162:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8e828793ed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:eaf4444c-5340-4caf-8764-d8e0f2a45321+5:15067:x1875062212041600:12345-192.168.201.121@tcp:104: 00010000:00010000:1.0:1788201580.026513:0:6162:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289561400/0x9ae3109cee9e210e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x7cf2114380cc24c2 expref: -99 pid: 42252 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201580.026582:0:6162:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1788201580.026589:0:6162:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8e828793ed80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:eaf4444c-5340-4caf-8764-d8e0f2a45321+5:15067:x1875062212041600:12345-192.168.201.121@tcp:104: Request processed in 94us (347us total) trans 0 rc 0/0 00000100:00100000:1.0:1788201580.026596:0:6162:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.201.121@tcp, seq: 94 00000100:00100000:3.0:1788201580.039643:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512680 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187541504:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.039668:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e2123 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2516 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.039698:0:42256:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e2123 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2516 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.039706:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e2123 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2516 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.039717:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e2123 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2516 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.039728:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e2123 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2516 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.039741:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e2123 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2516 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.039749:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e2123 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2516 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.039756:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e2123 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2516 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.039767:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8289512680 x1875062187541504/t81604378731(81604378731) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201580.039780:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289512680 x1875062187541504/t81604378731(81604378731) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.039805:0:42256:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e2123 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2516 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201580.039819:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e2123 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2516 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.039829:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/216 (0 0) 00000080:00200000:3.0:1788201580.039835:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201580.039857:0:42256:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-0, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:3.0:1788201580.039888:0:42256:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xd8:0x0](ffff8e8289e5c2c8) 00000080:00200000:3.0:1788201580.039967:0:42256:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xd8:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201580.039972:0:42256:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xd8:0x0](ffff8e8289e5c2c8) 00000080:00010000:3.0:1788201580.039975:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xd8:0x0](ffff8e8289e5c2c8) for lock 0x9ae3109cee9e2123 00000080:00010000:3.0:1788201580.039984:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xd8:0x0](ffff8e8289e5c2c8) 00000080:00200008:3.0:1788201580.039997:0:42256:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-0:[0x200006991:0xd8:0x0](ffff8e8289e5c2c8), flags 100102 00000002:00100000:3.0:1788201580.040017:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289512680 x1875062187541504/t81604378731(81604378731) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201580.040049:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2123 from it ffff8e82849b9d20 00010000:00010000:3.0:1788201580.040053:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e2123 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc2516 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.040063:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e2123 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc2516 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.040083:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xd8:0x0](ffff8e8289e5c2c8), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201580.040112:0:42256:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-0:[0x200006991:0xd8:0x0](ffff8e8289e5c2c8), flags 100002 00000002:00100000:3.0:1788201580.040155:0:42256:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8289512680 x1875062187541504/t81604378731(81604378731) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201580.040165:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201580.040171:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d717100 x1875062187541888/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201580.040190:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d717100 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187541888:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201580.040271:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2bb40 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.061961:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d717100 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187541888:192.168.201.121@tcp:35:createmany.0 00000002:00100000:3.0:1788201580.061997:0:42256:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8289512680 x1875062187541504/t81604378731(81604378731) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201580.062074:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.062091:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.062149:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e212a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.062159:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e212a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.062169:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e212a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.062186:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d717b80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187542016:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.062261:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2bb40 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.078079:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d717b80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187542016:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.078107:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e212a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc252b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.078120:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e212a lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc252b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.078131:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e212a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc252b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.078141:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e212a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc252b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.078158:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e212a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc252b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.078165:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e212a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc252b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.078171:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e212a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc252b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.078182:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d717b80 x1875062187542016/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.078196:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e212a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc252b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.078209:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201580.078216:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201580.078241:0:42256:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201580.078247:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e212a 00000080:00010000:3.0:1788201580.078252:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.078254:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e212a from it ffffaa1802033bf8 00010000:00010000:3.0:1788201580.078257:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e212a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc252b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.078265:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e212a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc252b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.078292:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201580.078352:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.078367:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.078375:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e212a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc252b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.078386:0:42256:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e212a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc252b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.078392:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201580.078397:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.078400:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e212a 00000080:00010000:3.0:1788201580.078404:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e212a from it ffffaa1802033b38 00010000:00010000:3.0:1788201580.078407:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e212a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc252b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.078413:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e212a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc252b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201580.078422:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201580.078427:0:42256:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-1, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e82828bc300, open_flags 8042, mode 81a4 opened 0 00000080:00200000:3.0:1788201580.078435:0:42256:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-1, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:3.0:1788201580.078464:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-1,[0x200006991:0xd9:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788201580.078485:0:42256:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e212a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc252b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.078532:0:42256:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e212a lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc252b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.078540:0:42256:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201580.078543:0:42256:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e212a lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc252b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.078578:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e2131 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.078584:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e2131 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201580.078595:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201580.078601:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d717b80 x1875062187542272/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201580.078610:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e2131 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.078629:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d717b80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187542272:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.078838:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2bb40 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.101886:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d717b80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187542272:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.101908:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e2131 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2547 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.101931:0:42256:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e2131 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2547 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.101943:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e2131 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2547 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.101954:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e2131 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2547 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.101964:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e2131 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2547 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.101977:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e2131 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2547 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.101986:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e2131 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2547 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.101991:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e2131 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2547 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.102001:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e828d717b80 x1875062187542272/t81604378733(81604378733) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201580.102013:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d717b80 x1875062187542272/t81604378733(81604378733) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.102025:0:42256:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e2131 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2547 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201580.102038:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e2131 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2547 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.102049:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/217 (0 0) 00000080:00200000:3.0:1788201580.102055:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201580.102066:0:42256:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-1, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:3.0:1788201580.102103:0:42256:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xd9:0x0](ffff8e8289e5d348) 00000080:00200000:3.0:1788201580.102177:0:42256:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xd9:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201580.102183:0:42256:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xd9:0x0](ffff8e8289e5d348) 00000080:00010000:3.0:1788201580.102186:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xd9:0x0](ffff8e8289e5d348) for lock 0x9ae3109cee9e2131 00000080:00010000:3.0:1788201580.102195:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xd9:0x0](ffff8e8289e5d348) 00000080:00200008:3.0:1788201580.102213:0:42256:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-1:[0x200006991:0xd9:0x0](ffff8e8289e5d348), flags 100102 00000002:00100000:3.0:1788201580.102234:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d717b80 x1875062187542272/t81604378733(81604378733) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201580.102266:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2131 from it ffff8e8290e1fba0 00010000:00010000:3.0:1788201580.102270:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e2131 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc2547 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.102276:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e2131 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc2547 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.102291:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xd9:0x0](ffff8e8289e5d348), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201580.102366:0:42256:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-1:[0x200006991:0xd9:0x0](ffff8e8289e5d348), flags 100002 00000002:00100000:3.0:1788201580.102408:0:42256:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e828d717b80 x1875062187542272/t81604378733(81604378733) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201580.102418:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201580.102424:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d715f80 x1875062187542528/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201580.102445:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d715f80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187542528:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201580.102538:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2bc00 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.106458:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d715f80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187542528:192.168.201.121@tcp:35:createmany.0 00000002:00100000:3.0:1788201580.106497:0:42256:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e828d717b80 x1875062187542272/t81604378733(81604378733) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201580.106572:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.106595:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.106652:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e2138 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.106662:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e2138 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.106672:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e2138 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.106690:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d714700 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187542656:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.106768:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2bc00 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.108736:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d714700 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187542656:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.108760:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e2138 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc255c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.108772:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e2138 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc255c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.108787:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e2138 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc255c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.108798:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e2138 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc255c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.108810:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e2138 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc255c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.108818:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e2138 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc255c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.108825:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e2138 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc255c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.108836:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d714700 x1875062187542656/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.108849:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e2138 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc255c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.108862:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201580.108867:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201580.108890:0:42256:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201580.108896:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2138 00000080:00010000:3.0:1788201580.108902:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.108905:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2138 from it ffffaa1802033bf8 00010000:00010000:3.0:1788201580.108909:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e2138 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc255c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.108918:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e2138 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc255c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.108939:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201580.108959:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.108972:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.108979:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e2138 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc255c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.108989:0:42256:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e2138 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc255c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.108996:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201580.109000:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.109003:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2138 00000080:00010000:3.0:1788201580.109007:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2138 from it ffffaa1802033b38 00010000:00010000:3.0:1788201580.109010:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e2138 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc255c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.109017:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e2138 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc255c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201580.109024:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201580.109029:0:42256:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-2, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e82828bd800, open_flags 8042, mode 81a4 opened 0 00000080:00200000:3.0:1788201580.109037:0:42256:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-2, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:3.0:1788201580.109057:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-2,[0x200006991:0xda:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788201580.109067:0:42256:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e2138 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc255c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.109106:0:42256:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e2138 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc255c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.109114:0:42256:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201580.109116:0:42256:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e2138 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc255c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.109148:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e213f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.109155:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e213f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201580.109167:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201580.109173:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d714700 x1875062187542912/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201580.109181:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e213f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.109199:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d714700 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187542912:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.109580:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2bc00 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.115144:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d714700 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187542912:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.115173:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e213f lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2578 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.115212:0:42256:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e213f lrc: 4/0,1 mode: --/PR res: [0x200006991:0xda:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2578 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.115222:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e213f lrc: 4/0,1 mode: --/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2578 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.115233:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e213f lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2578 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.115244:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e213f lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2578 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.115258:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e213f lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2578 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.115267:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e213f lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2578 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.115275:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e213f lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2578 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.115288:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e828d714700 x1875062187542912/t81604378735(81604378735) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201580.115302:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d714700 x1875062187542912/t81604378735(81604378735) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.115343:0:42256:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e213f lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2578 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201580.115357:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e213f lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2578 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.115368:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/218 (0 0) 00000080:00200000:3.0:1788201580.115375:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201580.115390:0:42256:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-2, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:3.0:1788201580.115421:0:42256:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xda:0x0](ffff8e8289e5db88) 00000100:00100000:3.0:1788201580.154063:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a5c00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187543936:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.154090:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e2154 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25be expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.154099:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e2154 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25be expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.154108:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e2154 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc25be expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.154115:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e2154 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25be expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.154124:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e2154 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc25be expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.154130:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e2154 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc25be expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.154138:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e2154 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc25be expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.154146:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d6a5c00 x1875062187543936/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.154155:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e2154 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25be expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.154164:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201580.154170:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201580.154193:0:42256:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201580.154197:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2154 00000080:00010000:3.0:1788201580.154200:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.154203:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2154 from it ffffaa1802033bf8 00010000:00010000:3.0:1788201580.154205:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e2154 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc25be expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.154211:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e2154 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25be expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.154229:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201580.154246:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.154258:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.154263:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e2154 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25be expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.154273:0:42256:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e2154 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25be expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.154276:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201580.154279:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.154281:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2154 00000080:00010000:3.0:1788201580.154283:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2154 from it ffffaa1802033b38 00010000:00010000:3.0:1788201580.154284:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e2154 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc25be expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.154289:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e2154 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25be expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201580.154293:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201580.154325:0:42256:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-4, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e82829c5e00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:3.0:1788201580.154332:0:42256:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-4, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:3.0:1788201580.154345:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-4,[0x200006991:0xdc:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788201580.154353:0:42256:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e2154 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc25be expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.154394:0:42256:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e2154 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc25be expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.154399:0:42256:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201580.154401:0:42256:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e2154 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc25be expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.154431:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e215b lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.154435:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e215b lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201580.154449:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201580.154453:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d716d80 x1875062187544192/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201580.154459:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e215b lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.154483:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d716d80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187544192:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.154621:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2bc00 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.163738:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716d80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187544192:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.163769:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e215b lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25da expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.163800:0:42256:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e215b lrc: 4/0,1 mode: --/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25da expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.163808:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e215b lrc: 4/0,1 mode: --/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25da expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.163819:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e215b lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc25da expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.163836:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e215b lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25da expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.163848:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e215b lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc25da expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.163857:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e215b lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc25da expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.163865:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e215b lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc25da expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.163876:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e828d716d80 x1875062187544192/t81604378739(81604378739) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201580.163889:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d716d80 x1875062187544192/t81604378739(81604378739) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.163902:0:42256:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e215b lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25da expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201580.163915:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e215b lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25da expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.163926:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/220 (0 0) 00000080:00200000:3.0:1788201580.163943:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201580.163956:0:42256:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-4, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:3.0:1788201580.163989:0:42256:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xdc:0x0](ffff8e8289e59988) 00000080:00200000:3.0:1788201580.164606:0:42256:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xdc:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201580.164614:0:42256:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xdc:0x0](ffff8e8289e59988) 00000080:00010000:3.0:1788201580.164620:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xdc:0x0](ffff8e8289e59988) for lock 0x9ae3109cee9e215b 00000080:00010000:3.0:1788201580.164632:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xdc:0x0](ffff8e8289e59988) 00000080:00200008:3.0:1788201580.164650:0:42256:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-4:[0x200006991:0xdc:0x0](ffff8e8289e59988), flags 100102 00000002:00100000:3.0:1788201580.164677:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d716d80 x1875062187544192/t81604378739(81604378739) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201580.164722:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e215b from it ffff8e8290e1f0c0 00010000:00010000:3.0:1788201580.164728:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e215b lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc25da expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.164738:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e215b lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc25da expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.164764:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xdc:0x0](ffff8e8289e59988), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201580.164795:0:42256:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-4:[0x200006991:0xdc:0x0](ffff8e8289e59988), flags 100002 00000002:00100000:3.0:1788201580.164836:0:42256:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e828d716d80 x1875062187544192/t81604378739(81604378739) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201580.164846:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201580.164851:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d715c00 x1875062187544448/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201580.164871:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d715c00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187544448:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201580.164948:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.174353:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d715c00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187544448:192.168.201.121@tcp:35:createmany.0 00000002:00100000:3.0:1788201580.174401:0:42256:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e828d716d80 x1875062187544192/t81604378739(81604378739) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201580.174516:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.174534:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.174596:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2162 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.174607:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2162 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.174619:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2162 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.174640:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d714380 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187544576:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.174711:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.180714:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d714380 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187544576:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.180741:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2162 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25ef expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.180754:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2162 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25ef expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.180764:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2162 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc25ef expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.180774:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2162 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25ef expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.180791:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2162 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc25ef expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.180798:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2162 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc25ef expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.180805:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2162 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc25ef expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.180815:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d714380 x1875062187544576/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.180828:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2162 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25ef expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.180840:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201580.180846:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201580.180872:0:42256:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201580.180878:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2162 00000080:00010000:3.0:1788201580.180884:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.180887:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2162 from it ffffaa1802033bf8 00010000:00010000:3.0:1788201580.180890:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2162 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc25ef expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.180899:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2162 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25ef expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.180928:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201580.180964:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.180979:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.180988:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2162 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25ef expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.181012:0:42256:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2162 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25ef expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.181020:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201580.181024:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.181027:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2162 00000080:00010000:3.0:1788201580.181031:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2162 from it ffffaa1802033b38 00010000:00010000:3.0:1788201580.181034:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2162 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc25ef expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.181042:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2162 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc25ef expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201580.181051:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201580.181056:0:42256:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-5, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e82828bc300, open_flags 8042, mode 81a4 opened 0 00000080:00200000:3.0:1788201580.181064:0:42256:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-5, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:3.0:1788201580.181084:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-5,[0x200006991:0xdd:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788201580.181096:0:42256:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2162 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc25ef expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.181144:0:42256:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2162 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc25ef expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.181152:0:42256:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201580.181155:0:42256:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2162 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc25ef expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.181190:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e2169 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.181197:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e2169 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201580.181208:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201580.181215:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d714380 x1875062187544832/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201580.181224:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e2169 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.181244:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d714380 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187544832:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.181671:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b300 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.190967:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d714380 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187544832:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.191007:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e2169 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc260b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.191036:0:42256:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e2169 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc260b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.191049:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e2169 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc260b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.191060:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e2169 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc260b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.191071:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e2169 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc260b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.191084:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e2169 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc260b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.191092:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e2169 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc260b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.191100:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e2169 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc260b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.191112:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e828d714380 x1875062187544832/t81604378741(81604378741) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201580.191125:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d714380 x1875062187544832/t81604378741(81604378741) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.191139:0:42256:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e2169 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc260b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201580.191152:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e2169 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc260b expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.191163:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/221 (0 0) 00000080:00200000:3.0:1788201580.191169:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201580.191181:0:42256:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-5, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:3.0:1788201580.191208:0:42256:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xdd:0x0](ffff8e8289e5cb08) 00000080:00200000:3.0:1788201580.191278:0:42256:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xdd:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201580.191283:0:42256:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xdd:0x0](ffff8e8289e5cb08) 00000080:00010000:3.0:1788201580.191286:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xdd:0x0](ffff8e8289e5cb08) for lock 0x9ae3109cee9e2169 00000080:00010000:3.0:1788201580.191554:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xdd:0x0](ffff8e8289e5cb08) 00000080:00200008:3.0:1788201580.191570:0:42256:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-5:[0x200006991:0xdd:0x0](ffff8e8289e5cb08), flags 100102 00000002:00100000:3.0:1788201580.191593:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d714380 x1875062187544832/t81604378741(81604378741) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201580.191629:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2169 from it ffff8e8290f36c60 00010000:00010000:3.0:1788201580.191635:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e2169 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc260b expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.191644:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e2169 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc260b expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.191665:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xdd:0x0](ffff8e8289e5cb08), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201580.191696:0:42256:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-5:[0x200006991:0xdd:0x0](ffff8e8289e5cb08), flags 100002 00000002:00100000:3.0:1788201580.191736:0:42256:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e828d714380 x1875062187544832/t81604378741(81604378741) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201580.191746:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201580.191752:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d716a00 x1875062187545088/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201580.191777:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d716a00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187545088:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201580.191849:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b840 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.200035:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716a00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187545088:192.168.201.121@tcp:35:createmany.0 00000002:00100000:3.0:1788201580.200072:0:42256:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e828d714380 x1875062187544832/t81604378741(81604378741) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201580.200150:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.200168:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.200225:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2170 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.200235:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2170 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.200246:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2170 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.200264:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d714000 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187545216:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201580.200761:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201580.202858:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d714000 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187545216:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201580.202885:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2170 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2620 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.202897:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2170 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2620 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.202909:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2170 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2620 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.202926:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2170 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2620 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.202938:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2170 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2620 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.202947:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2170 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2620 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.202954:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2170 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2620 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201580.202967:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d714000 x1875062187545216/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201580.202980:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2170 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2620 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.202992:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201580.203010:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201580.203035:0:42256:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201580.203041:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2170 00000080:00010000:1.0:1788201580.203047:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201580.203050:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2170 from it ffffaa1802033bf8 00010000:00010000:1.0:1788201580.203054:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2170 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2620 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.203069:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2170 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2620 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201580.203091:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201580.203113:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201580.203126:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201580.203134:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2170 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2620 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201580.203143:0:42256:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2170 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2620 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201580.203150:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201580.203154:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201580.203157:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2170 00000080:00010000:1.0:1788201580.203161:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2170 from it ffffaa1802033b38 00010000:00010000:1.0:1788201580.203163:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2170 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2620 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201580.203170:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2170 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2620 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201580.203178:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201580.203183:0:42256:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-6, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e82828bcc00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201580.203191:0:42256:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-6, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:1.0:1788201580.203207:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-6,[0x200006991:0xde:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201580.203217:0:42256:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2170 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc2620 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201580.203257:0:42256:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2170 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2620 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201580.203264:0:42256:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201580.203267:0:42256:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2170 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2620 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201580.203347:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992c00/0x9ae3109cee9e2177 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.203355:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992c00/0x9ae3109cee9e2177 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201580.203365:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201580.203381:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289510000 x1875062187545472/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201580.203393:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992c00/0x9ae3109cee9e2177 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201580.203415:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510000 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187545472:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201580.203541:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f5900 going to sleep for 11 seconds 00000100:00100000:1.0:1788201580.206806:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510000 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187545472:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201580.206832:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992c00/0x9ae3109cee9e2177 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc263c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.206862:0:42256:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992c00/0x9ae3109cee9e2177 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xde:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc263c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.206872:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992c00/0x9ae3109cee9e2177 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc263c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.206885:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992c00/0x9ae3109cee9e2177 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc263c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.206895:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992c00/0x9ae3109cee9e2177 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc263c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.206908:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992c00/0x9ae3109cee9e2177 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc263c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.206917:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992c00/0x9ae3109cee9e2177 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc263c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.206924:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992c00/0x9ae3109cee9e2177 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc263c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201580.206936:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8289510000 x1875062187545472/t81604378743(81604378743) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201580.206949:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289510000 x1875062187545472/t81604378743(81604378743) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201580.206962:0:42256:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992c00/0x9ae3109cee9e2177 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc263c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201580.206973:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992c00/0x9ae3109cee9e2177 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc263c expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.206984:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/222 (0 0) 00000080:00200000:1.0:1788201580.206990:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201580.207148:0:42256:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-6, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:1.0:1788201580.207175:0:42256:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xde:0x0](ffff8e8289c14b08) 00000080:00200000:1.0:1788201580.207259:0:42256:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xde:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201580.207265:0:42256:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xde:0x0](ffff8e8289c14b08) 00000080:00010000:1.0:1788201580.207268:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xde:0x0](ffff8e8289c14b08) for lock 0x9ae3109cee9e2177 00000080:00010000:1.0:1788201580.207277:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xde:0x0](ffff8e8289c14b08) 00000080:00200008:1.0:1788201580.207291:0:42256:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-6:[0x200006991:0xde:0x0](ffff8e8289c14b08), flags 100102 00000002:00100000:1.0:1788201580.207345:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289510000 x1875062187545472/t81604378743(81604378743) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201580.207506:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2177 from it ffff8e82849b9900 00010000:00010000:1.0:1788201580.207512:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992c00/0x9ae3109cee9e2177 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc263c expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.207521:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992c00/0x9ae3109cee9e2177 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc263c expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201580.207540:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xde:0x0](ffff8e8289c14b08), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201580.207572:0:42256:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-6:[0x200006991:0xde:0x0](ffff8e8289c14b08), flags 100002 00000002:00100000:1.0:1788201580.207619:0:42256:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8289510000 x1875062187545472/t81604378743(81604378743) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201580.207630:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201580.207634:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289511f80 x1875062187545728/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201580.207657:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511f80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187545728:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201580.207789:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f5000 going to sleep for 11 seconds 00000100:00100000:1.0:1788201580.216282:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511f80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187545728:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201580.226698:0:42256:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8289510000 x1875062187545472/t81604378743(81604378743) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201580.226786:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201580.226803:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201580.226861:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e217e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.226871:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e217e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.226881:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e217e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201580.226900:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513b80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187545856:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201580.227869:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f5000 going to sleep for 11 seconds 00000100:00100000:1.0:1788201580.230181:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289513b80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187545856:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201580.230207:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e217e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2651 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.230219:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e217e lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2651 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.230235:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e217e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2651 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.230244:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e217e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2651 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.230255:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e217e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2651 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.230263:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e217e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2651 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.230270:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e217e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2651 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201580.230282:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289513b80 x1875062187545856/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201580.230293:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e217e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2651 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.230337:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201580.230344:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201580.230380:0:42256:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201580.230387:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e217e 00000080:00010000:1.0:1788201580.230392:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201580.230395:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e217e from it ffffaa1802033bf8 00010000:00010000:1.0:1788201580.230399:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e217e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2651 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.230406:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e217e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2651 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201580.230429:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201580.230451:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201580.230464:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201580.230482:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e217e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2651 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201580.230493:0:42256:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e217e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2651 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201580.230499:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201580.230504:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201580.230507:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e217e 00000080:00010000:1.0:1788201580.230510:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e217e from it ffffaa1802033b38 00010000:00010000:1.0:1788201580.230514:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e217e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2651 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201580.230521:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e217e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2651 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201580.230529:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201580.230534:0:42256:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-7, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8291da8300, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201580.230541:0:42256:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-7, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:1.0:1788201580.230559:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-7,[0x200006991:0xdf:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201580.230570:0:42256:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e217e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc2651 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201580.230612:0:42256:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e217e lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2651 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201580.230620:0:42256:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201580.230623:0:42256:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e217e lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2651 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201580.230653:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e2185 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.230659:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e2185 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201580.230669:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201580.230677:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289513b80 x1875062187546112/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201580.230686:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e2185 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201580.230705:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513b80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187546112:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201580.230838:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f5000 going to sleep for 11 seconds 00000100:00100000:1.0:1788201580.236613:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289513b80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187546112:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201580.236641:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e2185 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc266d expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.236674:0:42256:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e2185 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc266d expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.236682:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e2185 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc266d expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.236694:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e2185 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc266d expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.236703:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e2185 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc266d expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.236716:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e2185 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc266d expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.236723:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e2185 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc266d expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.236730:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e2185 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc266d expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201580.236741:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8289513b80 x1875062187546112/t81604378745(81604378745) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201580.236753:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289513b80 x1875062187546112/t81604378745(81604378745) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201580.236766:0:42256:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e2185 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc266d expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201580.236778:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e2185 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc266d expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.236787:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/223 (0 0) 00000080:00200000:1.0:1788201580.236793:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201580.236805:0:42256:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-7, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:1.0:1788201580.236833:0:42256:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xdf:0x0](ffff8e8289c13a88) 00000080:00200000:1.0:1788201580.236897:0:42256:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xdf:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201580.236901:0:42256:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xdf:0x0](ffff8e8289c13a88) 00000080:00010000:1.0:1788201580.236904:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xdf:0x0](ffff8e8289c13a88) for lock 0x9ae3109cee9e2185 00000080:00010000:1.0:1788201580.236914:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xdf:0x0](ffff8e8289c13a88) 00000080:00200008:1.0:1788201580.236927:0:42256:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-7:[0x200006991:0xdf:0x0](ffff8e8289c13a88), flags 100102 00000002:00100000:1.0:1788201580.236944:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289513b80 x1875062187546112/t81604378745(81604378745) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201580.236979:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2185 from it ffff8e82849b9900 00010000:00010000:1.0:1788201580.236983:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e2185 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc266d expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.236992:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e2185 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc266d expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201580.237012:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xdf:0x0](ffff8e8289c13a88), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201580.237043:0:42256:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-7:[0x200006991:0xdf:0x0](ffff8e8289c13a88), flags 100002 00000002:00100000:1.0:1788201580.237082:0:42256:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8289513b80 x1875062187546112/t81604378745(81604378745) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201580.237098:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201580.237102:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289510a80 x1875062187546368/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201580.237122:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510a80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187546368:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201580.237194:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f5e40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201580.240271:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510a80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187546368:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201580.240342:0:42256:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8289513b80 x1875062187546112/t81604378745(81604378745) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201580.240420:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201580.240436:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201580.240498:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e218c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.240509:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e218c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.240518:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e218c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201580.240537:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510700 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187546496:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201580.240601:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f5e40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201580.243876:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510700 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187546496:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201580.243903:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e218c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2682 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.243920:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e218c lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2682 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.243930:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e218c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2682 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.243942:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e218c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2682 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.243954:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e218c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2682 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.243962:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e218c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2682 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.243970:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e218c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2682 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201580.243983:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289510700 x1875062187546496/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201580.244008:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e218c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2682 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.244021:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201580.244027:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201580.244053:0:42256:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201580.244060:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e218c 00000080:00010000:1.0:1788201580.244066:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201580.244069:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e218c from it ffffaa1802033bf8 00010000:00010000:1.0:1788201580.244073:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e218c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2682 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.244082:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e218c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2682 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201580.244107:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201580.244129:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201580.244142:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201580.244151:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e218c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2682 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201580.244162:0:42256:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e218c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2682 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201580.244169:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201580.244173:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201580.244176:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e218c 00000080:00010000:1.0:1788201580.244179:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e218c from it ffffaa1802033b38 00010000:00010000:1.0:1788201580.244183:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e218c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2682 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201580.244189:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e218c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2682 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201580.244215:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201580.244220:0:42256:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-8, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8291da8600, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201580.244228:0:42256:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-8, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:1.0:1788201580.244245:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-8,[0x200006991:0xe0:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201580.244255:0:42256:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e218c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc2682 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201580.244294:0:42256:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e218c lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2682 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201580.244301:0:42256:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201580.244305:0:42256:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e218c lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2682 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201580.244379:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e2193 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.244387:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e2193 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201580.244398:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201580.244404:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289510700 x1875062187546752/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201580.244414:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e2193 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201580.244432:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510700 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187546752:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201580.244551:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f5e40 going to sleep for 11 seconds 00000100:00100000:1.0:1788201580.250972:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510700 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187546752:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201580.251006:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e2193 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc269e expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.251035:0:42256:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e2193 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc269e expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.251045:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e2193 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc269e expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.251057:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e2193 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc269e expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.251068:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e2193 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc269e expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.251081:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e2193 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc269e expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.251089:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e2193 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc269e expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.251097:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e2193 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc269e expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201580.251109:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8289510700 x1875062187546752/t81604378747(81604378747) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201580.251124:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289510700 x1875062187546752/t81604378747(81604378747) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201580.251137:0:42256:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e2193 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc269e expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201580.251150:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e2193 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc269e expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.251160:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/224 (0 0) 00000080:00200000:1.0:1788201580.251166:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201580.251178:0:42256:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-8, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:1.0:1788201580.251212:0:42256:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xe0:0x0](ffff8e8289c121c8) 00000080:00200000:1.0:1788201580.251284:0:42256:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xe0:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201580.251289:0:42256:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xe0:0x0](ffff8e8289c121c8) 00000080:00010000:1.0:1788201580.251339:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xe0:0x0](ffff8e8289c121c8) for lock 0x9ae3109cee9e2193 00000080:00010000:1.0:1788201580.251349:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xe0:0x0](ffff8e8289c121c8) 00000080:00200008:1.0:1788201580.251364:0:42256:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-8:[0x200006991:0xe0:0x0](ffff8e8289c121c8), flags 100102 00000002:00100000:1.0:1788201580.251395:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289510700 x1875062187546752/t81604378747(81604378747) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201580.251429:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2193 from it ffff8e82849b9900 00010000:00010000:1.0:1788201580.251433:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e2193 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc269e expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.251441:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e2193 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc269e expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201580.251475:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xe0:0x0](ffff8e8289c121c8), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201580.251504:0:42256:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-8:[0x200006991:0xe0:0x0](ffff8e8289c121c8), flags 100002 00000002:00100000:1.0:1788201580.251546:0:42256:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8289510700 x1875062187546752/t81604378747(81604378747) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201580.251555:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201580.251561:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289512d80 x1875062187547008/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201580.251580:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512d80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187547008:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201580.251643:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f5600 going to sleep for 11 seconds 00000100:00100000:1.0:1788201580.259808:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512d80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187547008:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201580.259842:0:42256:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8289510700 x1875062187546752/t81604378747(81604378747) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201580.259915:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201580.259932:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201580.259987:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e219a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.260010:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e219a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.260020:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e219a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201580.260040:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512300 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187547136:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201580.260111:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f5600 going to sleep for 11 seconds 00000100:00100000:1.0:1788201580.262594:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512300 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187547136:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201580.262620:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e219a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc26b3 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.262633:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e219a lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc26b3 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.262644:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e219a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc26b3 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.262655:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e219a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc26b3 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.262667:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e219a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc26b3 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.262676:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e219a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc26b3 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.262683:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e219a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc26b3 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201580.262694:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289512300 x1875062187547136/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201580.262707:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e219a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc26b3 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.262724:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201580.262731:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201580.262755:0:42256:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201580.262762:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e219a 00000080:00010000:1.0:1788201580.262767:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201580.262770:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e219a from it ffffaa1802033bf8 00010000:00010000:1.0:1788201580.262774:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e219a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc26b3 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.262783:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e219a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc26b3 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201580.262806:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201580.262826:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201580.262839:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201580.262847:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e219a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc26b3 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201580.262857:0:42256:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e219a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc26b3 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201580.262865:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201580.262869:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201580.262871:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e219a 00000080:00010000:1.0:1788201580.262875:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e219a from it ffffaa1802033b38 00010000:00010000:1.0:1788201580.262878:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e219a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc26b3 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201580.262886:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e219a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc26b3 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201580.262894:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201580.262899:0:42256:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-9, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8291da8300, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201580.262908:0:42256:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-9, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:1.0:1788201580.262926:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-9,[0x200006991:0xe1:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201580.262937:0:42256:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e219a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc26b3 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201580.262984:0:42256:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e219a lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc26b3 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201580.262991:0:42256:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201580.262994:0:42256:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e219a lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc26b3 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201580.263091:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e21a1 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.263097:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e21a1 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201580.263108:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201580.263115:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289512300 x1875062187547392/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201580.263124:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e21a1 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201580.263143:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512300 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187547392:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201580.263276:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f5600 going to sleep for 11 seconds 00000100:00100000:1.0:1788201580.268684:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512300 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187547392:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201580.268713:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e21a1 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc26cf expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.268740:0:42256:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e21a1 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc26cf expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.268749:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e21a1 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc26cf expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.268760:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e21a1 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc26cf expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.268770:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e21a1 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc26cf expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.268785:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e21a1 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc26cf expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.268793:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e21a1 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc26cf expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.268800:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e21a1 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc26cf expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201580.268818:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8289512300 x1875062187547392/t81604378749(81604378749) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788201580.268831:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289512300 x1875062187547392/t81604378749(81604378749) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201580.268844:0:42256:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e21a1 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc26cf expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201580.268857:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e21a1 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc26cf expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.268868:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/225 (0 0) 00000080:00200000:1.0:1788201580.268874:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201580.268887:0:42256:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-9, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:1.0:1788201580.268914:0:42256:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xe1:0x0](ffff8e8289c17448) 00000080:00200000:1.0:1788201580.268985:0:42256:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xe1:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788201580.268990:0:42256:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xe1:0x0](ffff8e8289c17448) 00000080:00010000:1.0:1788201580.268993:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xe1:0x0](ffff8e8289c17448) for lock 0x9ae3109cee9e21a1 00000080:00010000:1.0:1788201580.269015:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xe1:0x0](ffff8e8289c17448) 00000080:00200008:1.0:1788201580.269030:0:42256:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-9:[0x200006991:0xe1:0x0](ffff8e8289c17448), flags 100102 00000002:00100000:1.0:1788201580.269049:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289512300 x1875062187547392/t81604378749(81604378749) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788201580.269083:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e21a1 from it ffff8e82849b9900 00010000:00010000:1.0:1788201580.269088:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e21a1 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc26cf expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201580.269101:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e21a1 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc26cf expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201580.269123:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xe1:0x0](ffff8e8289c17448), inode mode 81a4 mask 40 00000080:00200008:1.0:1788201580.269153:0:42256:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-9:[0x200006991:0xe1:0x0](ffff8e8289c17448), flags 100002 00000002:00100000:1.0:1788201580.269198:0:42256:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8289512300 x1875062187547392/t81604378749(81604378749) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788201580.269208:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788201580.269212:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289513100 x1875062187547648/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788201580.269232:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513100 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187547648:192.168.201.121@tcp:35:createmany.0 00000100:00100000:1.0:1788201580.269818:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f53c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201580.271992:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289513100 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187547648:192.168.201.121@tcp:35:createmany.0 00000002:00010000:3.0:1788201580.399795:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561800/0x9ae3109cee9e2203 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2826 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.399808:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/232 (0 0) 00000080:00200000:3.0:1788201580.399813:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201580.399825:0:42256:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-16, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:3.0:1788201580.399919:0:42256:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xe8:0x0](ffff8e8289e5ba88) 00000080:00200000:3.0:1788201580.399991:0:42256:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xe8:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201580.400043:0:42256:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xe8:0x0](ffff8e8289e5ba88) 00000080:00010000:3.0:1788201580.400047:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xe8:0x0](ffff8e8289e5ba88) for lock 0x9ae3109cee9e2203 00000080:00010000:3.0:1788201580.400057:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xe8:0x0](ffff8e8289e5ba88) 00000080:00200008:3.0:1788201580.400072:0:42256:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-16:[0x200006991:0xe8:0x0](ffff8e8289e5ba88), flags 100102 00000002:00100000:3.0:1788201580.400098:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d6a5180 x1875062187551872/t81604378763(81604378763) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201580.400131:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2203 from it ffff8e8290916360 00010000:00010000:3.0:1788201580.400136:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561800/0x9ae3109cee9e2203 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc2826 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.400145:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561800/0x9ae3109cee9e2203 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc2826 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.400166:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xe8:0x0](ffff8e8289e5ba88), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201580.400197:0:42256:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-16:[0x200006991:0xe8:0x0](ffff8e8289e5ba88), flags 100002 00000002:00100000:3.0:1788201580.400243:0:42256:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e828d6a5180 x1875062187551872/t81604378763(81604378763) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201580.400254:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201580.400260:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d715880 x1875062187552128/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201580.400279:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d715880 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187552128:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201580.400978:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b840 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.409217:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d715880 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187552128:192.168.201.121@tcp:35:createmany.0 00000002:00100000:3.0:1788201580.409253:0:42256:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e828d6a5180 x1875062187551872/t81604378763(81604378763) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201580.409364:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.409386:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.409442:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e220a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.409470:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e220a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.409480:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e220a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.409496:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d714a80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187552256:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.409562:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b840 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.417444:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d714a80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187552256:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.417480:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e220a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc283b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.417493:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e220a lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc283b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.417505:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e220a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc283b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.417525:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e220a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc283b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.417539:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e220a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc283b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.417547:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e220a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc283b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.417556:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e220a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc283b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.417568:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d714a80 x1875062187552256/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.417580:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e220a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc283b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.417594:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201580.417601:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201580.417763:0:42256:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201580.417771:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e220a 00000080:00010000:0.0:1788201580.417777:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201580.417781:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e220a from it ffffaa1802033bf8 00010000:00010000:0.0:1788201580.417785:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e220a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc283b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201580.417795:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e220a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc283b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201580.417819:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201580.418001:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201580.418018:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201580.418028:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e220a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc283b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201580.418040:0:42256:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e220a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc283b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201580.418048:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201580.418052:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201580.418056:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e220a 00000080:00010000:0.0:1788201580.418060:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e220a from it ffffaa1802033b38 00010000:00010000:0.0:1788201580.418064:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e220a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc283b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201580.418072:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e220a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc283b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201580.418081:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201580.418087:0:42256:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-17, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e82828bc300, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201580.418095:0:42256:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-17, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:0.0:1788201580.418115:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-17,[0x200006991:0xe9:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201580.418126:0:42256:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e220a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc283b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201580.418173:0:42256:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e220a lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc283b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201580.418181:0:42256:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201580.418185:0:42256:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e220a lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc283b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201580.418222:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e2211 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201580.418230:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e2211 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201580.418241:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201580.418249:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793df80 x1875062187552512/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201580.418259:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e2211 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201580.418278:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187552512:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201580.419682:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287a74240 going to sleep for 11 seconds 00000080:00200000:0.0:1788201580.429708:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xe9:0x0](ffff8e8289e421c8), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201580.429739:0:42256:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-17:[0x200006991:0xe9:0x0](ffff8e8289e421c8), flags 100002 00000002:00100000:0.0:1788201580.429776:0:42256:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e828793df80 x1875062187552512/t81604378765(81604378765) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201580.429785:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201580.429790:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793d880 x1875062187552768/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201580.429809:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187552768:192.168.201.121@tcp:35:createmany.0 00000100:00100000:0.0:1788201580.429883:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287a74780 going to sleep for 11 seconds 00000080:00200000:0.0:1788201580.438594:0:42256:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-18, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:0.0:1788201580.438614:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-18,[0x200006991:0xea:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201580.438625:0:42256:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e2218 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc286c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201580.438668:0:42256:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e2218 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc286c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201580.438675:0:42256:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201580.438678:0:42256:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e2218 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc286c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201580.438713:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e221f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201580.438718:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e221f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201580.438725:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201580.438731:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793ce00 x1875062187553152/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201580.438737:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e221f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201580.438752:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187553152:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201580.438874:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287a74600 going to sleep for 11 seconds 00000100:00100000:0.0:1788201580.483582:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289550e00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187554432:192.168.201.121@tcp:101:createmany.0 00010000:00010000:0.0:1788201580.483610:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561200/0x9ae3109cee9e223b lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc28ea expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201580.483640:0:42256:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561200/0x9ae3109cee9e223b lrc: 4/0,1 mode: --/PR res: [0x200006991:0xec:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc28ea expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201580.483649:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561200/0x9ae3109cee9e223b lrc: 4/0,1 mode: --/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc28ea expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201580.483660:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561200/0x9ae3109cee9e223b lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc28ea expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201580.483677:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561200/0x9ae3109cee9e223b lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc28ea expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201580.483690:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561200/0x9ae3109cee9e223b lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc28ea expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201580.483699:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561200/0x9ae3109cee9e223b lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc28ea expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201580.483707:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561200/0x9ae3109cee9e223b lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc28ea expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201580.483719:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8289550e00 x1875062187554432/t81604378771(81604378771) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788201580.483731:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289550e00 x1875062187554432/t81604378771(81604378771) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201580.483744:0:42256:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561200/0x9ae3109cee9e223b lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc28ea expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201580.483757:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561200/0x9ae3109cee9e223b lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc28ea expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201580.483767:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/236 (0 0) 00000080:00200000:0.0:1788201580.483773:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201580.483786:0:42256:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-20, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:0.0:1788201580.483817:0:42256:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xec:0x0](ffff8e8289c0db88) 00000080:00200000:0.0:1788201580.483890:0:42256:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xec:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201580.483896:0:42256:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xec:0x0](ffff8e8289c0db88) 00000080:00010000:0.0:1788201580.483899:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xec:0x0](ffff8e8289c0db88) for lock 0x9ae3109cee9e223b 00000080:00010000:0.0:1788201580.483907:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xec:0x0](ffff8e8289c0db88) 00000080:00200008:0.0:1788201580.483920:0:42256:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-20:[0x200006991:0xec:0x0](ffff8e8289c0db88), flags 100102 00000002:00100000:0.0:1788201580.483939:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289550e00 x1875062187554432/t81604378771(81604378771) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201580.483974:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e223b from it ffff8e8290916c00 00010000:00010000:0.0:1788201580.483980:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561200/0x9ae3109cee9e223b lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc28ea expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201580.483989:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561200/0x9ae3109cee9e223b lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc28ea expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201580.484011:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xec:0x0](ffff8e8289c0db88), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201580.484038:0:42256:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-20:[0x200006991:0xec:0x0](ffff8e8289c0db88), flags 100002 00000002:00100000:0.0:1788201580.484076:0:42256:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8289550e00 x1875062187554432/t81604378771(81604378771) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201580.484086:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201580.484091:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793c380 x1875062187554688/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201580.484111:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187554688:192.168.201.121@tcp:35:createmany.0 00000100:00100000:0.0:1788201580.484197:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287a74d80 going to sleep for 11 seconds 00000080:00200000:3.0:1788201580.496818:0:42256:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-21, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:3.0:1788201580.496895:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-21,[0x200006991:0xed:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788201580.496911:0:42256:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560400/0x9ae3109cee9e2242 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc28ff expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.496968:0:42256:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560400/0x9ae3109cee9e2242 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc28ff expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.496976:0:42256:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201580.496979:0:42256:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560400/0x9ae3109cee9e2242 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc28ff expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.497187:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e2249 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.497194:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e2249 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201580.497220:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201580.497230:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d716680 x1875062187555072/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201580.497238:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e2249 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.497258:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187555072:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.497743:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b840 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.502945:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187555072:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.502973:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e2249 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc291b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.503002:0:42256:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e2249 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xed:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc291b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.503011:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e2249 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc291b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.503022:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e2249 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc291b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.503033:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e2249 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc291b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.503046:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e2249 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc291b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.503060:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e2249 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc291b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.503067:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e2249 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc291b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.503079:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e828d716680 x1875062187555072/t81604378773(81604378773) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201580.503091:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d716680 x1875062187555072/t81604378773(81604378773) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.503105:0:42256:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e2249 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc291b expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201580.503118:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e2249 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc291b expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.503129:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/237 (0 0) 00000080:00200000:3.0:1788201580.503134:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201580.503147:0:42256:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-21, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:3.0:1788201580.503178:0:42256:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xed:0x0](ffff8e8289e5a1c8) 00000080:00200000:3.0:1788201580.503247:0:42256:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xed:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201580.503253:0:42256:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xed:0x0](ffff8e8289e5a1c8) 00000080:00010000:3.0:1788201580.503256:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xed:0x0](ffff8e8289e5a1c8) for lock 0x9ae3109cee9e2249 00000080:00010000:3.0:1788201580.503265:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xed:0x0](ffff8e8289e5a1c8) 00000080:00200008:3.0:1788201580.503278:0:42256:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-21:[0x200006991:0xed:0x0](ffff8e8289e5a1c8), flags 100102 00000002:00100000:3.0:1788201580.503299:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d716680 x1875062187555072/t81604378773(81604378773) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201580.503450:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2249 from it ffff8e82911e0360 00010000:00010000:3.0:1788201580.503455:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e2249 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc291b expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.503473:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e2249 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc291b expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.503493:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xed:0x0](ffff8e8289e5a1c8), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201580.503523:0:42256:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-21:[0x200006991:0xed:0x0](ffff8e8289e5a1c8), flags 100002 00000002:00100000:3.0:1788201580.503565:0:42256:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e828d716680 x1875062187555072/t81604378773(81604378773) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201580.503576:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201580.503581:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d714e00 x1875062187555328/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201580.503599:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d714e00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187555328:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201580.503718:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b180 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.507458:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d714e00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187555328:192.168.201.121@tcp:35:createmany.0 00000002:00100000:3.0:1788201580.507501:0:42256:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e828d716680 x1875062187555072/t81604378773(81604378773) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201580.507575:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.507592:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.507648:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e2250 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.507658:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e2250 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.507667:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e2250 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.507685:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d714a80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187555456:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.507760:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b180 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.512840:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d714a80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187555456:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.512924:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e2250 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2930 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.512947:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e2250 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2930 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.512959:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e2250 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2930 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.512970:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e2250 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2930 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.512984:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e2250 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2930 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.512993:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e2250 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2930 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.513006:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e2250 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2930 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.513018:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d714a80 x1875062187555456/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.513031:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e2250 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2930 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.513043:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201580.513050:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201580.513212:0:42256:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201580.513218:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2250 00000080:00010000:3.0:1788201580.513223:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.513226:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2250 from it ffffaa1802033bf8 00010000:00010000:3.0:1788201580.513229:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e2250 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2930 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.513236:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e2250 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2930 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.513258:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201580.513284:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.513341:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.513350:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e2250 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2930 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.513366:0:42256:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e2250 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2930 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.513373:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201580.513377:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.513380:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2250 00000080:00010000:3.0:1788201580.513384:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2250 from it ffffaa1802033b38 00010000:00010000:3.0:1788201580.513387:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e2250 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2930 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.513394:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e2250 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2930 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201580.513402:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201580.513407:0:42256:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-22, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e82828bca80, open_flags 8042, mode 81a4 opened 0 00000080:00200000:3.0:1788201580.513415:0:42256:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-22, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:3.0:1788201580.513539:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-22,[0x200006991:0xee:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788201580.513553:0:42256:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e2250 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc2930 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.513600:0:42256:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e2250 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2930 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.513608:0:42256:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201580.513612:0:42256:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e2250 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2930 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.513649:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e2257 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.513657:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e2257 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201580.513668:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201580.513675:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d714a80 x1875062187555712/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201580.513684:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e2257 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.513704:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d714a80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187555712:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.513921:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b180 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.536700:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d714a80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187555712:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.536727:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e2257 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc294c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.536758:0:42256:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e2257 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xee:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc294c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.536768:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e2257 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc294c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.536779:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e2257 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc294c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.536794:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e2257 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc294c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.536807:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e2257 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc294c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.536814:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e2257 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc294c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.536821:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e2257 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc294c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.536831:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e828d714a80 x1875062187555712/t81604378775(81604378775) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201580.536842:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d714a80 x1875062187555712/t81604378775(81604378775) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.536855:0:42256:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e2257 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc294c expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201580.536867:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e2257 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc294c expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.536876:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/238 (0 0) 00000080:00200000:3.0:1788201580.536882:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201580.536894:0:42256:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-22, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:3.0:1788201580.536922:0:42256:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xee:0x0](ffff8e8289e5e3c8) 00000080:00200000:3.0:1788201580.537016:0:42256:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xee:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201580.537021:0:42256:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xee:0x0](ffff8e8289e5e3c8) 00000080:00010000:3.0:1788201580.537024:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xee:0x0](ffff8e8289e5e3c8) for lock 0x9ae3109cee9e2257 00000080:00010000:3.0:1788201580.537034:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xee:0x0](ffff8e8289e5e3c8) 00000080:00200008:3.0:1788201580.537048:0:42256:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-22:[0x200006991:0xee:0x0](ffff8e8289e5e3c8), flags 100102 00000002:00100000:3.0:1788201580.537079:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d714a80 x1875062187555712/t81604378775(81604378775) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201580.537115:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2257 from it ffff8e82911e0360 00010000:00010000:3.0:1788201580.537120:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e2257 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc294c expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.537129:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e2257 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc294c expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.537150:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xee:0x0](ffff8e8289e5e3c8), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201580.537179:0:42256:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-22:[0x200006991:0xee:0x0](ffff8e8289e5e3c8), flags 100002 00000002:00100000:3.0:1788201580.537237:0:42256:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e828d714a80 x1875062187555712/t81604378775(81604378775) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201580.537250:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201580.537258:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d714000 x1875062187555968/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201580.537296:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d714000 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187555968:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201580.537590:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2be40 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.539804:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d714000 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187555968:192.168.201.121@tcp:35:createmany.0 00000002:00100000:3.0:1788201580.539834:0:42256:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e828d714a80 x1875062187555712/t81604378775(81604378775) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201580.539902:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.539918:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.539981:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e225e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.539991:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e225e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.540000:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e225e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.540016:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d716300 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187556096:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.540204:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2be40 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.543282:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716300 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187556096:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.543335:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e225e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2961 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.543347:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e225e lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2961 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.543358:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e225e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2961 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.543377:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e225e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2961 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.543393:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e225e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2961 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.543400:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e225e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2961 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.543407:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e225e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2961 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.543417:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d716300 x1875062187556096/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.543429:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e225e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2961 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.543451:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201580.543458:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201580.543492:0:42256:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201580.543498:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e225e 00000080:00010000:3.0:1788201580.543504:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.543507:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e225e from it ffffaa1802033bf8 00010000:00010000:3.0:1788201580.543510:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e225e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2961 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.543519:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e225e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2961 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.543548:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201580.543570:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.543585:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.543594:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e225e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2961 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.543604:0:42256:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e225e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2961 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.543611:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201580.543615:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.543618:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e225e 00000080:00010000:3.0:1788201580.543622:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e225e from it ffffaa1802033b38 00010000:00010000:3.0:1788201580.543625:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e225e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2961 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.543632:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e225e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2961 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201580.543639:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201580.543644:0:42256:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-23, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e82828bc000, open_flags 8042, mode 81a4 opened 0 00000080:00200000:3.0:1788201580.543652:0:42256:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-23, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:3.0:1788201580.543670:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-23,[0x200006991:0xef:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788201580.543682:0:42256:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e225e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc2961 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.543733:0:42256:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e225e lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2961 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.543740:0:42256:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201580.543743:0:42256:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e225e lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2961 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.543777:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e2265 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.543784:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e2265 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201580.543795:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201580.543802:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d716300 x1875062187556352/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201580.543811:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e2265 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.543829:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d716300 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187556352:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.543975:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2be40 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.548188:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716300 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187556352:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.548216:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e2265 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc297d expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.548245:0:42256:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e2265 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xef:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc297d expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.548259:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e2265 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc297d expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.548269:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e2265 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc297d expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.548281:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e2265 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc297d expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.548337:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e2265 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc297d expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.548347:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e2265 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc297d expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.548355:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e2265 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc297d expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.548376:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e828d716300 x1875062187556352/t81604378777(81604378777) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201580.548389:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d716300 x1875062187556352/t81604378777(81604378777) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.548403:0:42256:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e2265 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc297d expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201580.548416:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e2265 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc297d expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.548427:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/239 (0 0) 00000080:00200000:3.0:1788201580.548434:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201580.548456:0:42256:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-23, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:3.0:1788201580.548493:0:42256:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xef:0x0](ffff8e8289e5f448) 00000080:00200000:3.0:1788201580.548560:0:42256:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xef:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201580.548564:0:42256:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xef:0x0](ffff8e8289e5f448) 00000080:00010000:3.0:1788201580.548568:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xef:0x0](ffff8e8289e5f448) for lock 0x9ae3109cee9e2265 00000080:00010000:3.0:1788201580.548577:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xef:0x0](ffff8e8289e5f448) 00000080:00200008:3.0:1788201580.548590:0:42256:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-23:[0x200006991:0xef:0x0](ffff8e8289e5f448), flags 100102 00000002:00100000:3.0:1788201580.548609:0:42256:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d716300 x1875062187556352/t81604378777(81604378777) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201580.548643:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2265 from it ffff8e82911e0360 00010000:00010000:3.0:1788201580.548648:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e2265 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc297d expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.548658:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e2265 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc297d expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.548678:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xef:0x0](ffff8e8289e5f448), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201580.548706:0:42256:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-23:[0x200006991:0xef:0x0](ffff8e8289e5f448), flags 100002 00000002:00100000:3.0:1788201580.548748:0:42256:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e828d716300 x1875062187556352/t81604378777(81604378777) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201580.548758:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201580.548763:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d717800 x1875062187556608/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201580.548788:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d717800 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187556608:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201580.548863:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.551999:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d717800 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187556608:192.168.201.121@tcp:35:createmany.0 00000002:00100000:3.0:1788201580.552032:0:42256:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e828d716300 x1875062187556352/t81604378777(81604378777) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201580.552180:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.552198:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.552252:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e226c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.552263:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e226c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.552273:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e226c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.552302:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d715180 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187556736:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.552559:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.554804:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d715180 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187556736:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.554827:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e226c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2992 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.554839:0:42256:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e226c lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2992 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.554855:0:42256:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e226c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2992 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.554866:0:42256:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e226c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2992 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.554877:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e226c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2992 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.554885:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e226c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2992 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.554893:0:42256:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e226c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2992 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.554906:0:42256:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d715180 x1875062187556736/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.554918:0:42256:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e226c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2992 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.554929:0:42256:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201580.554936:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201580.554962:0:42256:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201580.554968:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e226c 00000080:00010000:3.0:1788201580.554973:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.554976:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e226c from it ffffaa1802033bf8 00010000:00010000:3.0:1788201580.554979:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e226c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2992 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.554988:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e226c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2992 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.555010:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201580.555030:0:42256:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.555043:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.555166:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e226c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2992 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.555179:0:42256:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e226c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2992 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.555187:0:42256:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201580.555192:0:42256:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.555195:0:42256:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e226c 00000080:00010000:3.0:1788201580.555199:0:42256:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e226c from it ffffaa1802033b38 00010000:00010000:3.0:1788201580.555202:0:42256:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e226c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2992 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.555210:0:42256:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e226c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2992 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201580.555219:0:42256:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201580.555225:0:42256:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-24, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e82828bca80, open_flags 8042, mode 81a4 opened 0 00000080:00200000:3.0:1788201580.555233:0:42256:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-24, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:3.0:1788201580.558594:0:42256:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-24,[0x200006991:0xf0:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788201580.558618:0:42256:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e226c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc2992 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.558686:0:42256:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e226c lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2992 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.558693:0:42256:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201580.558696:0:42256:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e226c lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2992 expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.558732:0:42256:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2273 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.558740:0:42256:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2273 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201580.558751:0:42256:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201580.558764:0:42256:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d715180 x1875062187556992/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201580.558773:0:42256:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2273 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42256 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.558792:0:42256:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d715180 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187556992:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.558943:0:42256:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.562509:0:42256:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d715180 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42256:1875062187556992:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.562535:0:42256:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2273 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc29ae expref: -99 pid: 42256 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:3.0:1788201580.676076:0:42264:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.676103:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.676122:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560600/0x9ae3109cee9e227a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc29c3 expref: -99 pid: 42260 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.676139:0:42264:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560600/0x9ae3109cee9e227a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc29c3 expref: -99 pid: 42260 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.676148:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201580.676155:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.676159:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e227a 00000080:00010000:3.0:1788201580.676167:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e227a from it ffffaa180202bbf8 00010000:00010000:3.0:1788201580.676172:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560600/0x9ae3109cee9e227a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc29c3 expref: -99 pid: 42260 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.676181:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560600/0x9ae3109cee9e227a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc29c3 expref: -99 pid: 42260 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201580.676196:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201580.676212:0:42264:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.676218:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.676225:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560600/0x9ae3109cee9e227a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc29c3 expref: -99 pid: 42260 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.676234:0:42264:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560600/0x9ae3109cee9e227a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc29c3 expref: -99 pid: 42260 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201580.676246:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201580.676251:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.676253:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e227a 00000080:00010000:3.0:1788201580.676257:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e227a from it ffffaa180202bb38 00010000:00010000:3.0:1788201580.676260:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560600/0x9ae3109cee9e227a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc29c3 expref: -99 pid: 42260 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.676268:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560600/0x9ae3109cee9e227a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc29c3 expref: -99 pid: 42260 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1788201580.676275:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201580.676280:0:42264:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-3-0, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8290c45e00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:3.0:1788201580.676299:0:42264:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-0, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:3.0:1788201580.676444:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-0,[0x200006991:0xf1:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788201580.676467:0:42264:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560600/0x9ae3109cee9e227a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc29c3 expref: -99 pid: 42260 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788201580.676537:0:42264:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560600/0x9ae3109cee9e227a lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc29c3 expref: -99 pid: 42260 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788201580.676545:0:42264:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201580.676548:0:42264:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560600/0x9ae3109cee9e227a lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc29c3 expref: -99 pid: 42260 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788201580.676583:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e228f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.676594:0:42264:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e228f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201580.676605:0:42264:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201580.676616:0:42264:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828953b480 x1875062187557888/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201580.676625:0:42264:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e228f lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.676645:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b480 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187557888:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.676780:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.680756:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b480 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187557888:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.680777:0:42264:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e228f lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2a17 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.680797:0:42264:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e228f lrc: 4/0,1 mode: --/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2a17 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.680803:0:42264:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e228f lrc: 4/0,1 mode: --/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2a17 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.680810:0:42264:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e228f lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2a17 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.680817:0:42264:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e228f lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2a17 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.680829:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e228f lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2a17 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.680835:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e228f lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2a17 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.680840:0:42264:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e228f lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2a17 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.680847:0:42264:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e828953b480 x1875062187557888/t81604378781(81604378781) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201580.680855:0:42264:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828953b480 x1875062187557888/t81604378781(81604378781) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.680863:0:42264:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e228f lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2a17 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201580.680871:0:42264:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e228f lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2a17 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.680878:0:42264:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/241 (0 0) 00000080:00200000:3.0:1788201580.680882:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201580.680893:0:42264:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-3-0, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:3.0:1788201580.680917:0:42264:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xf1:0x0](ffff8e8289e58908) 00000080:00200000:3.0:1788201580.681073:0:42264:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xf1:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201580.681078:0:42264:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xf1:0x0](ffff8e8289e58908) 00000080:00010000:3.0:1788201580.681082:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xf1:0x0](ffff8e8289e58908) for lock 0x9ae3109cee9e228f 00000080:00010000:3.0:1788201580.681091:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xf1:0x0](ffff8e8289e58908) 00000080:00200008:3.0:1788201580.681105:0:42264:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-3-0:[0x200006991:0xf1:0x0](ffff8e8289e58908), flags 100102 00000002:00100000:3.0:1788201580.681122:0:42264:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828953b480 x1875062187557888/t81604378781(81604378781) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201580.681149:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e228f from it ffff8e82911e07e0 00010000:00010000:3.0:1788201580.681152:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e228f lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc2a17 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.681159:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e228f lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc2a17 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.681175:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xf1:0x0](ffff8e8289e58908), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201580.681198:0:42264:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-3-0:[0x200006991:0xf1:0x0](ffff8e8289e58908), flags 100002 00000002:00100000:3.0:1788201580.681230:0:42264:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e828953b480 x1875062187557888/t81604378781(81604378781) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201580.681236:0:42264:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201580.681240:0:42264:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828953a300 x1875062187558144/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201580.681268:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a300 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187558144:192.168.201.121@tcp:35:createmany.0 00010000:00010000:3.0:1788201580.740711:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000002:00100000:3.0:1788201580.866593:0:42264:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e828d702a00 x1875062187564928/t81604378803(81604378803) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201580.866673:0:42264:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.866690:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.866745:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2330 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.866754:0:42264:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2330 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.866765:0:42264:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2330 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.866783:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b800 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187565312:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.866856:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b480 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.870014:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b800 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187565312:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.870039:0:42264:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2330 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c47 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.870060:0:42264:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2330 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c47 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.870073:0:42264:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2330 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2c47 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.870082:0:42264:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2330 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c47 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.870093:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2330 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2c47 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.870101:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2330 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2c47 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.870114:0:42264:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2330 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2c47 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.870124:0:42264:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828953b800 x1875062187565312/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.870137:0:42264:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2330 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c47 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.870149:0:42264:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201580.870155:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201580.870179:0:42264:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201580.870185:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2330 00000080:00010000:3.0:1788201580.870190:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.870193:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2330 from it ffffaa180202bbf8 00010000:00010000:3.0:1788201580.870196:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2330 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2c47 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.870205:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2330 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c47 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.870227:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201580.870249:0:42264:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.870263:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.870271:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2330 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c47 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.870533:0:42264:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2330 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c47 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.870542:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201580.870547:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.870550:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2330 00000080:00010000:3.0:1788201580.870554:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2330 from it ffffaa180202bb38 00010000:00010000:3.0:1788201580.870557:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2330 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2c47 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.870564:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2330 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c47 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201580.870572:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201580.870577:0:42264:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-3-12, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8290c44780, open_flags 8042, mode 81a4 opened 0 00000080:00200000:3.0:1788201580.870585:0:42264:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-12, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:3.0:1788201580.870603:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-12,[0x200006991:0xfd:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788201580.870614:0:42264:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2330 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc2c47 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.870660:0:42264:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2330 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2c47 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.870667:0:42264:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201580.870670:0:42264:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2330 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2c47 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.870707:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558200/0x9ae3109cee9e2337 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.870715:0:42264:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558200/0x9ae3109cee9e2337 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201580.870725:0:42264:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201580.870732:0:42264:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828953b800 x1875062187565568/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201580.870741:0:42264:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558200/0x9ae3109cee9e2337 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.870759:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b800 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187565568:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.870880:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8283d2b480 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.877583:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b800 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187565568:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.877608:0:42264:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558200/0x9ae3109cee9e2337 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c63 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.877638:0:42264:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558200/0x9ae3109cee9e2337 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c63 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.877648:0:42264:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558200/0x9ae3109cee9e2337 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c63 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.877660:0:42264:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558200/0x9ae3109cee9e2337 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2c63 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.877677:0:42264:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558200/0x9ae3109cee9e2337 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c63 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.877690:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558200/0x9ae3109cee9e2337 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2c63 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.877699:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558200/0x9ae3109cee9e2337 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2c63 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.877705:0:42264:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558200/0x9ae3109cee9e2337 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2c63 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.877717:0:42264:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e828953b800 x1875062187565568/t81604378805(81604378805) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201580.877728:0:42264:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828953b800 x1875062187565568/t81604378805(81604378805) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.877740:0:42264:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558200/0x9ae3109cee9e2337 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c63 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201580.877752:0:42264:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558200/0x9ae3109cee9e2337 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c63 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.877762:0:42264:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/253 (0 0) 00000080:00200000:3.0:1788201580.877767:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201580.877778:0:42264:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-3-12, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:3.0:1788201580.877809:0:42264:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xfd:0x0](ffff8e8289e5ec08) 00000080:00200000:3.0:1788201580.877883:0:42264:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xfd:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201580.877889:0:42264:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xfd:0x0](ffff8e8289e5ec08) 00000080:00010000:3.0:1788201580.877892:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xfd:0x0](ffff8e8289e5ec08) for lock 0x9ae3109cee9e2337 00000080:00010000:3.0:1788201580.877902:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xfd:0x0](ffff8e8289e5ec08) 00000080:00200008:3.0:1788201580.877916:0:42264:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-3-12:[0x200006991:0xfd:0x0](ffff8e8289e5ec08), flags 100102 00000002:00100000:3.0:1788201580.877937:0:42264:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828953b800 x1875062187565568/t81604378805(81604378805) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201580.877971:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2337 from it ffff8e82911e07e0 00010000:00010000:3.0:1788201580.877975:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558200/0x9ae3109cee9e2337 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc2c63 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.877986:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558200/0x9ae3109cee9e2337 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc2c63 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.878021:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xfd:0x0](ffff8e8289e5ec08), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201580.878119:0:42264:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-3-12:[0x200006991:0xfd:0x0](ffff8e8289e5ec08), flags 100002 00000002:00100000:3.0:1788201580.878160:0:42264:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e828953b800 x1875062187565568/t81604378805(81604378805) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201580.878170:0:42264:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201580.878175:0:42264:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289538e00 x1875062187565824/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201580.878195:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538e00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187565824:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201580.878268:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82923ed9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.880926:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538e00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187565824:192.168.201.121@tcp:35:createmany.0 00000002:00100000:3.0:1788201580.880957:0:42264:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e828953b800 x1875062187565568/t81604378805(81604378805) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201580.881184:0:42264:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.881200:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.881250:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e233e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.881260:0:42264:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e233e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.881271:0:42264:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e233e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.881463:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539180 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187565952:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.881569:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82923ed9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.884099:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539180 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187565952:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.884123:0:42264:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e233e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c78 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.884136:0:42264:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e233e lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c78 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.884148:0:42264:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e233e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2c78 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.884159:0:42264:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e233e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c78 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.884176:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e233e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2c78 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.884185:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e233e lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2c78 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.884193:0:42264:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e233e lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2c78 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.884204:0:42264:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289539180 x1875062187565952/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.884215:0:42264:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e233e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c78 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.884228:0:42264:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201580.884234:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201580.884258:0:42264:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201580.884264:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e233e 00000080:00010000:3.0:1788201580.884269:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.884271:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e233e from it ffffaa180202bbf8 00010000:00010000:3.0:1788201580.884275:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e233e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2c78 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.884597:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e233e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c78 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.884625:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201580.884648:0:42264:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.884662:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.884672:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e233e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c78 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.884682:0:42264:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e233e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c78 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.884688:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201580.884693:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.884695:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e233e 00000080:00010000:3.0:1788201580.884699:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e233e from it ffffaa180202bb38 00010000:00010000:3.0:1788201580.884703:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e233e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2c78 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.884711:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e233e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c78 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201580.884719:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201580.884724:0:42264:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-3-13, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8290c44180, open_flags 8042, mode 81a4 opened 0 00000080:00200000:3.0:1788201580.884731:0:42264:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-13, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:3.0:1788201580.884748:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-13,[0x200006991:0xfe:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788201580.884758:0:42264:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e233e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc2c78 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.884807:0:42264:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e233e lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2c78 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.884814:0:42264:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201580.884817:0:42264:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e233e lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2c78 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.884850:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2345 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.884857:0:42264:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2345 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201580.884867:0:42264:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201580.884874:0:42264:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289539180 x1875062187566208/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201580.884883:0:42264:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2345 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.884900:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539180 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187566208:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.885636:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82923ed9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.889523:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539180 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187566208:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.889548:0:42264:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2345 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c94 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.889576:0:42264:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2345 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c94 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.889591:0:42264:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2345 lrc: 4/0,1 mode: --/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c94 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.889602:0:42264:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2345 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2c94 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.889612:0:42264:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2345 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c94 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.889624:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2345 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2c94 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.889633:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2345 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2c94 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.889640:0:42264:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2345 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2c94 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.889652:0:42264:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8289539180 x1875062187566208/t81604378807(81604378807) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201580.889664:0:42264:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289539180 x1875062187566208/t81604378807(81604378807) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.889677:0:42264:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2345 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c94 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201580.889689:0:42264:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2345 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2c94 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.889700:0:42264:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/254 (0 0) 00000080:00200000:3.0:1788201580.889706:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201580.889718:0:42264:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-3-13, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:3.0:1788201580.889743:0:42264:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0xfe:0x0](ffff8e8289e59148) 00000080:00200000:3.0:1788201580.889807:0:42264:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xfe:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201580.889812:0:42264:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0xfe:0x0](ffff8e8289e59148) 00000080:00010000:3.0:1788201580.889816:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xfe:0x0](ffff8e8289e59148) for lock 0x9ae3109cee9e2345 00000080:00010000:3.0:1788201580.889825:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xfe:0x0](ffff8e8289e59148) 00000080:00200008:3.0:1788201580.889839:0:42264:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-3-13:[0x200006991:0xfe:0x0](ffff8e8289e59148), flags 100102 00000002:00100000:3.0:1788201580.889857:0:42264:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289539180 x1875062187566208/t81604378807(81604378807) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201580.889891:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2345 from it ffff8e82911e07e0 00010000:00010000:3.0:1788201580.889896:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2345 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc2c94 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.889906:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2345 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc2c94 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.889926:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0xfe:0x0](ffff8e8289e59148), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201580.889954:0:42264:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-3-13:[0x200006991:0xfe:0x0](ffff8e8289e59148), flags 100002 00000002:00100000:3.0:1788201580.889991:0:42264:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8289539180 x1875062187566208/t81604378807(81604378807) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201580.890224:0:42264:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201580.890230:0:42264:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828953a680 x1875062187566464/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201580.890256:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187566464:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201580.890526:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82923ed900 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.892931:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187566464:192.168.201.121@tcp:35:createmany.0 00000002:00100000:3.0:1788201580.892964:0:42264:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8289539180 x1875062187566208/t81604378807(81604378807) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201580.893198:0:42264:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.893218:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.893272:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e234c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.893491:0:42264:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e234c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.893504:0:42264:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e234c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.893521:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187566592:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.893616:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82923ed900 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.895685:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187566592:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.895703:0:42264:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e234c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ca9 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.895714:0:42264:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e234c lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ca9 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.895731:0:42264:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e234c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ca9 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.895740:0:42264:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e234c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ca9 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.895751:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e234c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ca9 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.895759:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e234c lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ca9 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.895767:0:42264:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e234c lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ca9 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.895777:0:42264:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289539880 x1875062187566592/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.895790:0:42264:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e234c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ca9 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.895800:0:42264:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201580.895805:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201580.895827:0:42264:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201580.895832:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e234c 00000080:00010000:3.0:1788201580.895836:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.895839:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e234c from it ffffaa180202bbf8 00010000:00010000:3.0:1788201580.895843:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e234c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ca9 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.895852:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e234c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ca9 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.895870:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201580.895890:0:42264:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.895901:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.895908:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e234c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ca9 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.895917:0:42264:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e234c lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ca9 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.895924:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201580.895927:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.895930:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e234c 00000080:00010000:3.0:1788201580.895933:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e234c from it ffffaa180202bb38 00010000:00010000:3.0:1788201580.895935:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e234c lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ca9 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.895942:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e234c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ca9 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201580.895950:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201580.895954:0:42264:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-3-14, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8290c44780, open_flags 8042, mode 81a4 opened 0 00000080:00200000:3.0:1788201580.895961:0:42264:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-14, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:3.0:1788201580.895977:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-14,[0x200006991:0xff:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788201580.895987:0:42264:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e234c lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc2ca9 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.896239:0:42264:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e234c lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2ca9 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.896248:0:42264:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201580.896250:0:42264:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e234c lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2ca9 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.896278:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e2353 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.896466:0:42264:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e2353 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201580.896477:0:42264:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201580.896485:0:42264:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289539880 x1875062187566848/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201580.896496:0:42264:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e2353 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.911613:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d702d80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187567232:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.911642:0:42264:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65a00/0x9ae3109cee9e235a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2cda expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.911656:0:42264:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65a00/0x9ae3109cee9e235a lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2cda expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.911674:0:42264:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65a00/0x9ae3109cee9e235a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2cda expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.911685:0:42264:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65a00/0x9ae3109cee9e235a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2cda expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.911699:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65a00/0x9ae3109cee9e235a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2cda expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.911708:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65a00/0x9ae3109cee9e235a lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2cda expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.911715:0:42264:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65a00/0x9ae3109cee9e235a lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2cda expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.911726:0:42264:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d702d80 x1875062187567232/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.911740:0:42264:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65a00/0x9ae3109cee9e235a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2cda expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.911751:0:42264:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201580.911758:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201580.911785:0:42264:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201580.911790:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e235a 00000080:00010000:3.0:1788201580.911795:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.911798:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e235a from it ffffaa180202bbf8 00010000:00010000:3.0:1788201580.911802:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65a00/0x9ae3109cee9e235a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2cda expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.911811:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65a00/0x9ae3109cee9e235a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2cda expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.911834:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201580.911861:0:42264:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.911875:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.911883:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65a00/0x9ae3109cee9e235a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2cda expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.911893:0:42264:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65a00/0x9ae3109cee9e235a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2cda expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.911898:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201580.911903:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.911905:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e235a 00000080:00010000:3.0:1788201580.911908:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e235a from it ffffaa180202bb38 00010000:00010000:3.0:1788201580.911911:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65a00/0x9ae3109cee9e235a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2cda expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.911916:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65a00/0x9ae3109cee9e235a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2cda expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201580.911923:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201580.911927:0:42264:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-3-15, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e828cb92900, open_flags 8042, mode 81a4 opened 0 00000080:00200000:3.0:1788201580.911936:0:42264:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-15, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:3.0:1788201580.911955:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-15,[0x200006991:0x100:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788201580.911965:0:42264:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65a00/0x9ae3109cee9e235a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc2cda expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00100000:3.0:1788201580.962729:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d703100 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187568768:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.962754:0:42264:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65400/0x9ae3109cee9e237d lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2d58 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.962784:0:42264:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65400/0x9ae3109cee9e237d lrc: 4/0,1 mode: --/PR res: [0x200006991:0x102:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2d58 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.962794:0:42264:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65400/0x9ae3109cee9e237d lrc: 4/0,1 mode: --/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2d58 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.962805:0:42264:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65400/0x9ae3109cee9e237d lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2d58 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.962816:0:42264:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65400/0x9ae3109cee9e237d lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2d58 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.962829:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65400/0x9ae3109cee9e237d lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2d58 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.962838:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65400/0x9ae3109cee9e237d lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2d58 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.962844:0:42264:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65400/0x9ae3109cee9e237d lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2d58 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.962856:0:42264:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e828d703100 x1875062187568768/t81604378815(81604378815) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201580.962868:0:42264:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d703100 x1875062187568768/t81604378815(81604378815) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.962881:0:42264:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65400/0x9ae3109cee9e237d lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2d58 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201580.962894:0:42264:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65400/0x9ae3109cee9e237d lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2d58 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.962903:0:42264:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/258 (0 0) 00000080:00200000:3.0:1788201580.962909:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201580.962922:0:42264:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-3-17, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:3.0:1788201580.962966:0:42264:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0x102:0x0](ffff8e8289e5b248) 00000080:00200000:3.0:1788201580.963036:0:42264:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x102:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201580.963041:0:42264:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0x102:0x0](ffff8e8289e5b248) 00000080:00010000:3.0:1788201580.963157:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x102:0x0](ffff8e8289e5b248) for lock 0x9ae3109cee9e237d 00000080:00010000:3.0:1788201580.963168:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x102:0x0](ffff8e8289e5b248) 00000080:00200008:3.0:1788201580.963183:0:42264:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-3-17:[0x200006991:0x102:0x0](ffff8e8289e5b248), flags 100102 00000002:00100000:3.0:1788201580.963204:0:42264:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d703100 x1875062187568768/t81604378815(81604378815) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201580.963237:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e237d from it ffff8e8289485c60 00010000:00010000:3.0:1788201580.963242:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65400/0x9ae3109cee9e237d lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc2d58 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.963256:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65400/0x9ae3109cee9e237d lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc2d58 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.963277:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0x102:0x0](ffff8e8289e5b248), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201580.963326:0:42264:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-3-17:[0x200006991:0x102:0x0](ffff8e8289e5b248), flags 100002 00000002:00100000:3.0:1788201580.964023:0:42264:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e828d703100 x1875062187568768/t81604378815(81604378815) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201580.964035:0:42264:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201580.964041:0:42264:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828953aa00 x1875062187569024/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201580.964131:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953aa00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187569024:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201580.964216:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82923ed3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.967154:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953aa00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187569024:192.168.201.121@tcp:35:createmany.0 00000002:00100000:3.0:1788201580.967184:0:42264:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e828d703100 x1875062187568768/t81604378815(81604378815) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201580.967259:0:42264:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.967277:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.967459:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e2384 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.967470:0:42264:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e2384 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.967480:0:42264:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e2384 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.967499:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187569152:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.967840:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82923ed3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.977292:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187569152:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.977349:0:42264:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e2384 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2d6d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.977591:0:42264:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e2384 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2d6d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.977604:0:42264:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e2384 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2d6d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.977616:0:42264:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e2384 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2d6d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.977630:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e2384 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2d6d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.977639:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e2384 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2d6d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.977647:0:42264:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e2384 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2d6d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.977660:0:42264:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289538380 x1875062187569152/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201596 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.977681:0:42264:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e2384 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2d6d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.977696:0:42264:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201580.977703:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201580.977730:0:42264:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201580.977736:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2384 00000080:00010000:3.0:1788201580.977741:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.977745:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2384 from it ffffaa180202bbf8 00010000:00010000:3.0:1788201580.977749:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e2384 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2d6d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.977758:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e2384 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2d6d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.977782:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201580.977804:0:42264:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201580.977819:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201580.977828:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e2384 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2d6d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.977838:0:42264:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e2384 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2d6d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.977845:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201580.977849:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201580.977852:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2384 00000080:00010000:3.0:1788201580.977856:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2384 from it ffffaa180202bb38 00010000:00010000:3.0:1788201580.977859:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e2384 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2d6d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201580.977867:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e2384 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2d6d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201580.977875:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201580.977880:0:42264:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-3-18, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8290c44180, open_flags 8042, mode 81a4 opened 0 00000080:00200000:3.0:1788201580.977889:0:42264:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-18, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:3.0:1788201580.977907:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-18,[0x200006991:0x103:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788201580.977981:0:42264:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e2384 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc2d6d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.978042:0:42264:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e2384 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2d6d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.978049:0:42264:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201580.978052:0:42264:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e2384 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2d6d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201580.978086:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e238b lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.978094:0:42264:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e238b lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201580.978104:0:42264:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201580.978112:0:42264:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289538380 x1875062187569408/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201580.978120:0:42264:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e238b lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201580.978138:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187569408:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201580.978278:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82923ed3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201580.990160:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187569408:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201580.990187:0:42264:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e238b lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2d89 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.990216:0:42264:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e238b lrc: 4/0,1 mode: --/PR res: [0x200006991:0x103:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2d89 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.990226:0:42264:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e238b lrc: 4/0,1 mode: --/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2d89 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.990237:0:42264:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e238b lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2d89 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.990248:0:42264:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e238b lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2d89 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.990262:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e238b lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2d89 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.990271:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e238b lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2d89 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.990426:0:42264:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e238b lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2d89 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201580.990448:0:42264:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8289538380 x1875062187569408/t81604378817(81604378817) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201580.990471:0:42264:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289538380 x1875062187569408/t81604378817(81604378817) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201580.990486:0:42264:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e238b lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2d89 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201580.990501:0:42264:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e238b lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2d89 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.990512:0:42264:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/259 (0 0) 00000080:00200000:3.0:1788201580.990519:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201580.990532:0:42264:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-3-18, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:3.0:1788201580.990562:0:42264:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0x103:0x0](ffff8e8289e5aa08) 00000080:00200000:3.0:1788201580.990631:0:42264:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x103:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201580.990636:0:42264:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0x103:0x0](ffff8e8289e5aa08) 00000080:00010000:3.0:1788201580.990640:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x103:0x0](ffff8e8289e5aa08) for lock 0x9ae3109cee9e238b 00000080:00010000:3.0:1788201580.990648:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x103:0x0](ffff8e8289e5aa08) 00000080:00200008:3.0:1788201580.990663:0:42264:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-3-18:[0x200006991:0x103:0x0](ffff8e8289e5aa08), flags 100102 00000002:00100000:3.0:1788201580.990682:0:42264:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289538380 x1875062187569408/t81604378817(81604378817) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201580.990721:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e238b from it ffff8e82911e04e0 00010000:00010000:3.0:1788201580.990725:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e238b lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc2d89 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201580.990735:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e238b lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc2d89 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201580.990755:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0x103:0x0](ffff8e8289e5aa08), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201580.990785:0:42264:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-3-18:[0x200006991:0x103:0x0](ffff8e8289e5aa08), flags 100002 00000002:00100000:3.0:1788201580.990827:0:42264:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8289538380 x1875062187569408/t81604378817(81604378817) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201580.990837:0:42264:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201580.990856:0:42264:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289539c00 x1875062187569664/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201580.990876:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539c00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187569664:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201580.991181:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82923eda80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201581.000295:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539c00 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187569664:192.168.201.121@tcp:35:createmany.0 00000080:00200000:0.0:1788201581.004680:0:42264:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-19, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:0.0:1788201581.004702:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-19,[0x200006991:0x104:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201581.004715:0:42264:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65a00/0x9ae3109cee9e2392 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc2d9e expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201581.004766:0:42264:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65a00/0x9ae3109cee9e2392 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2d9e expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201581.004774:0:42264:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201581.004777:0:42264:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65a00/0x9ae3109cee9e2392 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2d9e expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201581.004811:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1000/0x9ae3109cee9e2399 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201581.004819:0:42264:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1000/0x9ae3109cee9e2399 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201581.004831:0:42264:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201581.004839:0:42264:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793f480 x1875062187570048/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201581.004849:0:42264:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1000/0x9ae3109cee9e2399 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201581.004868:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187570048:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201581.005177:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287a74c00 going to sleep for 11 seconds 00000080:00200000:0.0:1788201581.019246:0:42264:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201581.019265:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201581.019279:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23a0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2dcf expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201581.019298:0:42264:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23a0 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2dcf expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201581.019305:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201581.019329:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201581.019332:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e23a0 00000080:00010000:0.0:1788201581.019338:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23a0 from it ffffaa180202bb38 00010000:00010000:0.0:1788201581.019341:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23a0 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2dcf expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201581.019349:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23a0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2dcf expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201581.019514:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201581.019520:0:42264:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-3-20, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e828cb93500, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788201581.019528:0:42264:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-20, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:0.0:1788201581.019550:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-20,[0x200006991:0x105:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788201581.019561:0:42264:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23a0 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc2dcf expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201581.019618:0:42264:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23a0 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2dcf expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201581.019626:0:42264:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788201581.019629:0:42264:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23a0 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2dcf expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201581.019665:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e23a7 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201581.019670:0:42264:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e23a7 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788201581.019678:0:42264:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788201581.019685:0:42264:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793fb80 x1875062187570688/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788201581.019691:0:42264:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e23a7 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201581.019707:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187570688:192.168.201.121@tcp:101:createmany.0 00000100:00100000:0.0:1788201581.019815:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287a74540 going to sleep for 11 seconds 00010000:00010000:2.0:1788201581.029468:0:42264:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23ae lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201581.029488:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d702300 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187571072:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201581.029610:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c900 going to sleep for 11 seconds 00000100:00100000:2.0:1788201581.032329:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d702300 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187571072:192.168.201.121@tcp:101:createmany.0 00010000:00010000:2.0:1788201581.032447:0:42264:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23ae lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e00 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.032468:0:42264:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23ae lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e00 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.032480:0:42264:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23ae lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e00 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.032492:0:42264:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23ae lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e00 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.032505:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23ae lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e00 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.032512:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23ae lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e00 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.032519:0:42264:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23ae lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e00 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201581.032531:0:42264:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d702300 x1875062187571072/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201597 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201581.032550:0:42264:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23ae lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e00 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.032562:0:42264:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201581.032569:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201581.032594:0:42264:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201581.032601:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e23ae 00000080:00010000:2.0:1788201581.032607:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201581.032610:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23ae from it ffffaa180202bbf8 00010000:00010000:2.0:1788201581.032614:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23ae lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e00 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.032624:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23ae lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e00 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201581.032646:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201581.032668:0:42264:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201581.032680:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201581.032688:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23ae lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e00 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201581.032700:0:42264:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23ae lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e00 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201581.032706:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201581.032710:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201581.032713:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e23ae 00000080:00010000:2.0:1788201581.032716:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23ae from it ffffaa180202bb38 00010000:00010000:2.0:1788201581.032720:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23ae lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e00 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201581.032727:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23ae lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e00 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201581.032736:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201581.032741:0:42264:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-3-21, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e828cb93680, open_flags 8042, mode 81a4 opened 0 00000080:00200000:2.0:1788201581.032748:0:42264:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-21, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:2.0:1788201581.032764:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-21,[0x200006991:0x106:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:2.0:1788201581.032775:0:42264:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23ae lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc2e00 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201581.032817:0:42264:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23ae lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2e00 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201581.032824:0:42264:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201581.032827:0:42264:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23ae lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2e00 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201581.032857:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64000/0x9ae3109cee9e23b5 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.032864:0:42264:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64000/0x9ae3109cee9e23b5 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201581.032873:0:42264:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201581.032880:0:42264:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d702300 x1875062187571328/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201581.032890:0:42264:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64000/0x9ae3109cee9e23b5 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201581.032906:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d702300 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187571328:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201581.033260:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c900 going to sleep for 11 seconds 00000100:00100000:3.0:1788201581.037259:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d702300 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187571328:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201581.037284:0:42264:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64000/0x9ae3109cee9e23b5 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e1c expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.037343:0:42264:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64000/0x9ae3109cee9e23b5 lrc: 4/0,1 mode: --/PR res: [0x200006991:0x106:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e1c expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.037424:0:42264:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64000/0x9ae3109cee9e23b5 lrc: 4/0,1 mode: --/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e1c expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.037435:0:42264:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64000/0x9ae3109cee9e23b5 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e1c expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.037446:0:42264:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64000/0x9ae3109cee9e23b5 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e1c expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.037468:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64000/0x9ae3109cee9e23b5 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e1c expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.037476:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64000/0x9ae3109cee9e23b5 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e1c expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.037483:0:42264:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64000/0x9ae3109cee9e23b5 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e1c expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201581.037494:0:42264:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e828d702300 x1875062187571328/t81604378823(81604378823) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201597 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201581.037507:0:42264:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d702300 x1875062187571328/t81604378823(81604378823) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201597 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201581.037520:0:42264:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64000/0x9ae3109cee9e23b5 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e1c expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201581.037532:0:42264:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64000/0x9ae3109cee9e23b5 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e1c expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.037543:0:42264:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/262 (0 0) 00000080:00200000:3.0:1788201581.037549:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201581.037561:0:42264:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-3-21, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:3.0:1788201581.037588:0:42264:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0x106:0x0](ffff8e8289e4a1c8) 00000080:00200000:3.0:1788201581.037658:0:42264:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x106:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788201581.037663:0:42264:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0x106:0x0](ffff8e8289e4a1c8) 00000080:00010000:3.0:1788201581.037666:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x106:0x0](ffff8e8289e4a1c8) for lock 0x9ae3109cee9e23b5 00000080:00010000:3.0:1788201581.037675:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x106:0x0](ffff8e8289e4a1c8) 00000080:00200008:3.0:1788201581.037687:0:42264:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-3-21:[0x200006991:0x106:0x0](ffff8e8289e4a1c8), flags 100102 00000002:00100000:3.0:1788201581.037705:0:42264:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d702300 x1875062187571328/t81604378823(81604378823) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201597 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788201581.037738:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23b5 from it ffff8e8289485a80 00010000:00010000:3.0:1788201581.037744:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64000/0x9ae3109cee9e23b5 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc2e1c expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.037752:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64000/0x9ae3109cee9e23b5 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc2e1c expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201581.037771:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0x106:0x0](ffff8e8289e4a1c8), inode mode 81a4 mask 40 00000080:00200008:3.0:1788201581.037799:0:42264:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-3-21:[0x200006991:0x106:0x0](ffff8e8289e4a1c8), flags 100002 00000002:00100000:3.0:1788201581.037841:0:42264:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e828d702300 x1875062187571328/t81604378823(81604378823) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201597 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788201581.037850:0:42264:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788201581.037856:0:42264:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828953ad80 x1875062187571584/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788201581.037879:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953ad80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187571584:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201581.038162:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82923ed6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201581.041487:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953ad80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187571584:192.168.201.121@tcp:35:createmany.0 00000002:00100000:3.0:1788201581.041523:0:42264:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e828d702300 x1875062187571328/t81604378823(81604378823) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201597 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788201581.041601:0:42264:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201581.041618:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201581.041670:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e23bc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.041680:0:42264:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e23bc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.041689:0:42264:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e23bc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201581.041706:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538700 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187571712:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201581.041810:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82923ed6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201581.044882:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538700 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187571712:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201581.044906:0:42264:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e23bc lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e31 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.044919:0:42264:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e23bc lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e31 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.044937:0:42264:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e23bc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e31 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.045166:0:42264:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e23bc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e31 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.045180:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e23bc lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e31 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.045189:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e23bc lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e31 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.045198:0:42264:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e23bc lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e31 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201581.045209:0:42264:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289538700 x1875062187571712/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201597 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201581.045222:0:42264:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e23bc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e31 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.045234:0:42264:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201581.045241:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201581.045265:0:42264:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201581.045271:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e23bc 00000080:00010000:3.0:1788201581.045276:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201581.045422:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23bc from it ffffaa180202bbf8 00010000:00010000:3.0:1788201581.045427:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e23bc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e31 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.045437:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e23bc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e31 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201581.045468:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201581.045492:0:42264:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201581.045505:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201581.045514:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e23bc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e31 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201581.045525:0:42264:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e23bc lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e31 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201581.045531:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201581.045536:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201581.045538:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e23bc 00000080:00010000:3.0:1788201581.045542:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23bc from it ffffaa180202bb38 00010000:00010000:3.0:1788201581.045545:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e23bc lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e31 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201581.045553:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e23bc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e31 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201581.045561:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201581.045566:0:42264:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-3-22, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8290c44300, open_flags 8042, mode 81a4 opened 0 00000080:00200000:3.0:1788201581.045573:0:42264:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-22, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:3.0:1788201581.045590:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-22,[0x200006991:0x107:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788201581.045601:0:42264:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e23bc lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc2e31 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201581.045648:0:42264:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e23bc lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2e31 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201581.045655:0:42264:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788201581.045658:0:42264:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558c00/0x9ae3109cee9e23bc lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2e31 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201581.045692:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e23c3 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.045699:0:42264:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e23c3 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788201581.045709:0:42264:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788201581.045716:0:42264:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289538700 x1875062187571968/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788201581.045726:0:42264:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e23c3 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201581.045743:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538700 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187571968:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201581.045928:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82923ed6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201581.050156:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538700 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187571968:192.168.201.121@tcp:101:createmany.0 00010000:00010000:3.0:1788201581.050181:0:42264:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e23c3 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e4d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.050217:0:42264:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e23c3 lrc: 4/0,1 mode: --/PR res: [0x200006991:0x107:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e4d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.050227:0:42264:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e23c3 lrc: 4/0,1 mode: --/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e4d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.050238:0:42264:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e23c3 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e4d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.050249:0:42264:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e23c3 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e4d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.050264:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e23c3 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e4d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.050273:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e23c3 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e4d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.050529:0:42264:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e23c3 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e4d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201581.050543:0:42264:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8289538700 x1875062187571968/t81604378825(81604378825) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201597 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788201581.050556:0:42264:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289538700 x1875062187571968/t81604378825(81604378825) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201597 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201581.050569:0:42264:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e23c3 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e4d expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201581.050640:0:42264:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e23c3 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e4d expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201581.050654:0:42264:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/263 (0 0) 00000080:00200000:0.0:1788201581.050661:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201581.050673:0:42264:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-3-22, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:0.0:1788201581.050703:0:42264:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0x107:0x0](ffff8e8289c0d348) 00000080:00200000:0.0:1788201581.050771:0:42264:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x107:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788201581.050776:0:42264:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0x107:0x0](ffff8e8289c0d348) 00000080:00010000:0.0:1788201581.050780:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x107:0x0](ffff8e8289c0d348) for lock 0x9ae3109cee9e23c3 00000080:00010000:0.0:1788201581.050788:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x107:0x0](ffff8e8289c0d348) 00000080:00200008:0.0:1788201581.050801:0:42264:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-3-22:[0x200006991:0x107:0x0](ffff8e8289c0d348), flags 100102 00000002:00100000:0.0:1788201581.050820:0:42264:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289538700 x1875062187571968/t81604378825(81604378825) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201597 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788201581.050855:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23c3 from it ffff8e82911e03c0 00010000:00010000:0.0:1788201581.050860:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e23c3 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc2e4d expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201581.050869:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e23c3 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc2e4d expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201581.050889:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0x107:0x0](ffff8e8289c0d348), inode mode 81a4 mask 40 00000080:00200008:0.0:1788201581.050916:0:42264:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-3-22:[0x200006991:0x107:0x0](ffff8e8289c0d348), flags 100002 00000002:00100000:0.0:1788201581.050965:0:42264:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8289538700 x1875062187571968/t81604378825(81604378825) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201597 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788201581.050975:0:42264:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788201581.050980:0:42264:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793f100 x1875062187572224/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788201581.051012:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187572224:192.168.201.121@tcp:35:createmany.0 00000100:00100000:0.0:1788201581.051241:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8287a74000 going to sleep for 11 seconds 00000100:00100000:1.0:1788201581.053968:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187572224:192.168.201.121@tcp:35:createmany.0 00000002:00100000:1.0:1788201581.054002:0:42264:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8289538700 x1875062187571968/t81604378825(81604378825) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201597 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788201581.054175:0:42264:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201581.054193:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201581.054243:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e23ca lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201581.054252:0:42264:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e23ca lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201581.054267:0:42264:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e23ca lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201581.054295:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511500 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187572352:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201581.054529:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f53c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201581.057097:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511500 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187572352:192.168.201.121@tcp:101:createmany.0 00010000:00010000:1.0:1788201581.057122:0:42264:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e23ca lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e62 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201581.057134:0:42264:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e23ca lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e62 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201581.057146:0:42264:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e23ca lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e62 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201581.057156:0:42264:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e23ca lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e62 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201581.057168:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e23ca lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e62 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201581.057175:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e23ca lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e62 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201581.057183:0:42264:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e23ca lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e62 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201581.057199:0:42264:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289511500 x1875062187572352/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201597 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201581.057212:0:42264:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e23ca lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e62 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201581.057224:0:42264:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201581.057230:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201581.057255:0:42264:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201581.057262:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e23ca 00000080:00010000:1.0:1788201581.057267:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201581.057270:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23ca from it ffffaa180202bbf8 00010000:00010000:1.0:1788201581.057274:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e23ca lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e62 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201581.057292:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e23ca lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e62 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201581.057349:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201581.057381:0:42264:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201581.057395:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201581.057404:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e23ca lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e62 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201581.057414:0:42264:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e23ca lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e62 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201581.057425:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201581.057429:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201581.057432:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e23ca 00000080:00010000:1.0:1788201581.057436:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23ca from it ffffaa180202bb38 00010000:00010000:1.0:1788201581.057439:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e23ca lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e62 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201581.057447:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e23ca lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e62 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201581.057463:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201581.057469:0:42264:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-3-23, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e8282990180, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788201581.057478:0:42264:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-23, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:1.0:1788201581.057497:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-23,[0x200006991:0x108:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788201581.057507:0:42264:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e23ca lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc2e62 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201581.057555:0:42264:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e23ca lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2e62 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201581.057562:0:42264:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788201581.057565:0:42264:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e23ca lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2e62 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201581.057597:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993800/0x9ae3109cee9e23d1 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201581.057608:0:42264:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993800/0x9ae3109cee9e23d1 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788201581.057617:0:42264:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788201581.057624:0:42264:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289511500 x1875062187572608/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788201581.057633:0:42264:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993800/0x9ae3109cee9e23d1 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201581.057651:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511500 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187572608:192.168.201.121@tcp:101:createmany.0 00000100:00100000:1.0:1788201581.057771:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f53c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201581.063442:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187572736:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201581.063533:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187572992:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201581.063559:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:eaf4444c-5340-4caf-8764-d8e0f2a45321:2402:1875062187573248:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201581.063579:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:eaf4444c-5340-4caf-8764-d8e0f2a45321:2402:1875062187573504:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201581.063604:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 4 async RPCs [0->1] 00000100:00100000:1.0:1788201581.063610:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187572864:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201581.063629:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187573120:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201581.063646:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:eaf4444c-5340-4caf-8764-d8e0f2a45321:2402:1875062187573376:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201581.063663:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:eaf4444c-5340-4caf-8764-d8e0f2a45321:2402:1875062187573632:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201581.064927:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511500 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187572608:192.168.201.121@tcp:101:createmany.0 00010000:00010000:2.0:1788201581.064958:0:42264:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993800/0x9ae3109cee9e23d1 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e7e expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.064985:0:42264:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993800/0x9ae3109cee9e23d1 lrc: 4/0,1 mode: --/PR res: [0x200006991:0x108:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e7e expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.064993:0:42264:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993800/0x9ae3109cee9e23d1 lrc: 4/0,1 mode: --/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e7e expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.065014:0:42264:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993800/0x9ae3109cee9e23d1 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e7e expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.065025:0:42264:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993800/0x9ae3109cee9e23d1 lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e7e expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.065038:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993800/0x9ae3109cee9e23d1 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e7e expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.065151:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993800/0x9ae3109cee9e23d1 lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e7e expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.065166:0:42264:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993800/0x9ae3109cee9e23d1 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e7e expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201581.065179:0:42264:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e8289511500 x1875062187572608/t81604378827(81604378827) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201597 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201581.065191:0:42264:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289511500 x1875062187572608/t81604378827(81604378827) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201597 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201581.065206:0:42264:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993800/0x9ae3109cee9e23d1 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e7e expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201581.065218:0:42264:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993800/0x9ae3109cee9e23d1 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e7e expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.065229:0:42264:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/264 (0 0) 00000080:00200000:2.0:1788201581.065235:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201581.065248:0:42264:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-3-23, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:2.0:1788201581.065278:0:42264:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0x108:0x0](ffff8e8292759148) 00000080:00200000:2.0:1788201581.065382:0:42264:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x108:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788201581.065387:0:42264:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0x108:0x0](ffff8e8292759148) 00000080:00010000:2.0:1788201581.065391:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x108:0x0](ffff8e8292759148) for lock 0x9ae3109cee9e23d1 00000080:00010000:2.0:1788201581.065399:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x108:0x0](ffff8e8292759148) 00000080:00200008:2.0:1788201581.065412:0:42264:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-3-23:[0x200006991:0x108:0x0](ffff8e8292759148), flags 100102 00000002:00100000:2.0:1788201581.065430:0:42264:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e8289511500 x1875062187572608/t81604378827(81604378827) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201597 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788201581.065469:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23d1 from it ffff8e82849b9900 00010000:00010000:2.0:1788201581.065474:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993800/0x9ae3109cee9e23d1 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc2e7e expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.065483:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993800/0x9ae3109cee9e23d1 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc2e7e expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201581.065504:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0x108:0x0](ffff8e8292759148), inode mode 81a4 mask 40 00000080:00200008:2.0:1788201581.065532:0:42264:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-3-23:[0x200006991:0x108:0x0](ffff8e8292759148), flags 100002 00000002:00100000:2.0:1788201581.065570:0:42264:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e8289511500 x1875062187572608/t81604378827(81604378827) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201597 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201581.065580:0:42264:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201581.065584:0:42264:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d701f80 x1875062187573888/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788201581.065601:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d701f80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187573888:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201581.065651:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c900 going to sleep for 11 seconds 00000100:00100000:2.0:1788201581.071522:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d701f80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187573888:192.168.201.121@tcp:35:createmany.0 00000002:00100000:2.0:1788201581.071555:0:42264:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e8289511500 x1875062187572608/t81604378827(81604378827) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201597 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201581.071631:0:42264:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201581.071647:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201581.071702:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23d8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.071717:0:42264:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23d8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.071728:0:42264:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23d8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201581.071746:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d702d80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187574016:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201581.071802:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c900 going to sleep for 11 seconds 00000100:00100000:1.0:1788201581.072067:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289513480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187572864:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201581.072080:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187573120:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201581.072090:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:eaf4444c-5340-4caf-8764-d8e0f2a45321:2402:1875062187573376:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201581.072099:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:eaf4444c-5340-4caf-8764-d8e0f2a45321:2402:1875062187573632:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201581.072109:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187572992:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201581.072120:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:eaf4444c-5340-4caf-8764-d8e0f2a45321:2402:1875062187573248:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201581.072131:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:eaf4444c-5340-4caf-8764-d8e0f2a45321:2402:1875062187573504:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201581.076249:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289513800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187572736:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201581.076774:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d702d80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187574016:192.168.201.121@tcp:101:createmany.0 00010000:00010000:2.0:1788201581.076795:0:42264:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23d8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e93 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.076809:0:42264:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23d8 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e93 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.076821:0:42264:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23d8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e93 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.076832:0:42264:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23d8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e93 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.076844:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23d8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e93 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.076853:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23d8 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e93 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.076867:0:42264:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23d8 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e93 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201581.076879:0:42264:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d702d80 x1875062187574016/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201597 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201581.076893:0:42264:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23d8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e93 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.076904:0:42264:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201581.076911:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201581.076937:0:42264:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201581.076954:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e23d8 00000080:00010000:2.0:1788201581.076959:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201581.076962:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23d8 from it ffffaa180202bbf8 00010000:00010000:2.0:1788201581.076966:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23d8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e93 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.076974:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23d8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e93 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201581.077008:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201581.077032:0:42264:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201581.077216:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201581.077226:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23d8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e93 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201581.077242:0:42264:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23d8 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e93 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201581.077249:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201581.077253:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201581.077257:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e23d8 00000080:00010000:2.0:1788201581.077262:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23d8 from it ffffaa180202bb38 00010000:00010000:2.0:1788201581.077265:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23d8 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2e93 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201581.077271:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23d8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2e93 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201581.077285:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201581.077290:0:42264:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f12a.replay-vbr-3-24, dir=[0x200000007:0x1:0x0](ffff8e8292531988), file ffff8e828cb93500, open_flags 8042, mode 81a4 opened 0 00000080:00200000:2.0:1788201581.077298:0:42264:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-24, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000002:00010000:2.0:1788201581.077339:0:42264:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-24,[0x200006991:0x109:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:2.0:1788201581.077350:0:42264:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23d8 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc2e93 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201581.077764:0:42264:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23d8 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2e93 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201581.077771:0:42264:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201581.077774:0:42264:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64400/0x9ae3109cee9e23d8 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2e93 expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201581.077813:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23df lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.077820:0:42264:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23df lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788201581.077830:0:42264:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788201581.077839:0:42264:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d702d80 x1875062187574272/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788201581.077848:0:42264:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23df lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42264 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201581.077867:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d702d80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187574272:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201581.078009:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c900 going to sleep for 11 seconds 00000100:00100000:2.0:1788201581.082359:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d702d80 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187574272:192.168.201.121@tcp:101:createmany.0 00010000:00010000:2.0:1788201581.082385:0:42264:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23df lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2eaf expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.082412:0:42264:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23df lrc: 4/0,1 mode: --/PR res: [0x200006991:0x109:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2eaf expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.082421:0:42264:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23df lrc: 4/0,1 mode: --/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2eaf expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.082433:0:42264:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23df lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2eaf expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.082448:0:42264:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23df lrc: 4/0,1 mode: PR/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2eaf expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.082470:0:42264:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23df lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2eaf expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.082478:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23df lrc: 5/1,1 mode: PR/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2eaf expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.082486:0:42264:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23df lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2eaf expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201581.082498:0:42264:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff8e828d702d80 x1875062187574272/t81604378829(81604378829) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201597 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788201581.082511:0:42264:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d702d80 x1875062187574272/t81604378829(81604378829) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 664/664 e 0 to 0 dl 1788201597 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201581.082525:0:42264:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23df lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2eaf expref: -99 pid: 42264 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201581.082538:0:42264:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23df lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2eaf expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.082548:0:42264:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/265 (0 0) 00000080:00200000:2.0:1788201581.082555:0:42264:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201581.082567:0:42264:0:(namei.c:1628:ll_create_it()) VFS Op:name=f12a.replay-vbr-3-24, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=open|creat 00000080:00200000:2.0:1788201581.082596:0:42264:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200006991:0x109:0x0](ffff8e829275f448) 00000080:00200000:2.0:1788201581.082661:0:42264:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x109:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788201581.082666:0:42264:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x200006991:0x109:0x0](ffff8e829275f448) 00000080:00010000:2.0:1788201581.082669:0:42264:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x109:0x0](ffff8e829275f448) for lock 0x9ae3109cee9e23df 00000080:00010000:2.0:1788201581.082678:0:42264:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x109:0x0](ffff8e829275f448) 00000080:00200008:2.0:1788201581.082691:0:42264:0:(file.c:1090:ll_file_open()) START file f12a.replay-vbr-3-24:[0x200006991:0x109:0x0](ffff8e829275f448), flags 100102 00000002:00100000:2.0:1788201581.082711:0:42264:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff8e828d702d80 x1875062187574272/t81604378829(81604378829) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201597 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788201581.082745:0:42264:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23df from it ffff8e8289485960 00010000:00010000:2.0:1788201581.082750:0:42264:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23df lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc2eaf expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.082760:0:42264:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23df lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc2eaf expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201581.082778:0:42264:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200006991:0x109:0x0](ffff8e829275f448), inode mode 81a4 mask 40 00000080:00200008:2.0:1788201581.082803:0:42264:0:(file.c:479:ll_file_release()) START file f12a.replay-vbr-3-24:[0x200006991:0x109:0x0](ffff8e829275f448), flags 100002 00000002:00100000:2.0:1788201581.082840:0:42264:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8e828d702d80 x1875062187574272/t81604378829(81604378829) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201597 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788201581.082850:0:42264:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788201581.082855:0:42264:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894d7480 x1875062187574528/t0(0) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788201581.082874:0:42264:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7480 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187574528:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201581.082957:0:42264:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828799c0c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201581.085989:0:42264:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7480 pname:cluuid:pid:xid:nid:opc:job createmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42264:1875062187574528:192.168.201.121@tcp:35:createmany.0 00000002:00100000:2.0:1788201581.086038:0:42264:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff8e828d702d80 x1875062187574272/t81604378829(81604378829) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201597 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1788201581.282897:0:42275:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8e8292531988), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201581.282909:0:42275:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201581.282932:0:42275:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201581.282990:0:42275:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.283001:0:42275:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.283011:0:42275:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42275 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201581.283034:0:42275:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job lsof:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42275:1875062187574656:192.168.201.121@tcp:101:lsof.0 00000100:00100000:3.0:1788201581.283193:0:42275:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82923ed6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201581.285899:0:42275:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job lsof:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42275:1875062187574656:192.168.201.121@tcp:101:lsof.0 00010000:00010000:3.0:1788201581.285925:0:42275:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.285937:0:42275:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.285947:0:42275:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.285958:0:42275:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.285976:0:42275:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.285984:0:42275:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.285991:0:42275:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201581.286014:0:42275:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289539f80 x1875062187574656/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201597 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lsof.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201581.286027:0:42275:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.286104:0:42275:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201581.286112:0:42275:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201581.286137:0:42275:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201581.286143:0:42275:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e23e6 00000080:00010000:3.0:1788201581.286148:0:42275:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201581.286151:0:42275:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23e6 from it ffffaa180205bcb8 00010000:00010000:3.0:1788201581.286154:0:42275:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.286163:0:42275:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788201581.288443:0:42275:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8e8289c080c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201581.288466:0:42275:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8),name=/ 00000002:00010000:3.0:1788201581.288487:0:42275:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201581.288540:0:42275:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.288550:0:42275:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.288561:0:42275:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42275 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201581.288585:0:42275:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job lsof:eaf4444c-5340-4caf-8764-d8e0f2a45321:42275:1875062187574784:192.168.201.121@tcp:101:lsof.0 00000100:00100000:3.0:1788201581.288667:0:42275:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82923ed6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201581.291262:0:42275:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job lsof:eaf4444c-5340-4caf-8764-d8e0f2a45321:42275:1875062187574784:192.168.201.121@tcp:101:lsof.0 00010000:00010000:3.0:1788201581.291467:0:42275:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.291481:0:42275:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.291494:0:42275:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.291505:0:42275:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.291523:0:42275:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.291531:0:42275:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.291539:0:42275:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201581.291551:0:42275:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289539f80 x1875062187574784/t0(0) o101->lustre-MDT0000-mdc-ffff8e82879e2000@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201597 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lsof.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201581.291566:0:42275:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201581.291578:0:42275:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e829232b000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201581.291584:0:42275:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201581.291701:0:42275:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201581.291710:0:42275:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c080c8) for lock 0x9ae3109cee9e23ed 00000080:00010000:2.0:1788201581.291716:0:42275:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c080c8) 00000080:00010000:2.0:1788201581.291719:0:42275:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23ed from it ffffaa180205bcb8 00010000:00010000:2.0:1788201581.291723:0:42275:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201581.291732:0:42275:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201581.771749:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201582.791336:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788201582.993199:0:42283:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8e8289c080c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201582.993215:0:42283:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8),name=/ 00000002:00010000:3.0:1788201582.993429:0:42283:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201582.993454:0:42283:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201582.993473:0:42283:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201582.993482:0:42283:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201582.993493:0:42283:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c080c8) 00000080:00010000:3.0:1788201582.993496:0:42283:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c080c8) for lock 0x9ae3109cee9e23ed 00000080:00010000:3.0:1788201582.993507:0:42283:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23ed from it ffffaa18020a3cb8 00010000:00010000:3.0:1788201582.993511:0:42283:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201582.993521:0:42283:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200008:3.0:1788201582.993672:0:42283:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8e8289c080c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788201582.993677:0:42283:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8),name=/ 00000002:00010000:3.0:1788201582.993685:0:42283:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201582.993691:0:42283:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201582.993699:0:42283:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201582.993705:0:42283:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201582.993709:0:42283:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c080c8) 00000080:00010000:3.0:1788201582.993711:0:42283:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8289c080c8) for lock 0x9ae3109cee9e23ed 00000080:00010000:3.0:1788201582.993715:0:42283:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23ed from it ffffaa18020a3cb8 00010000:00010000:3.0:1788201582.993718:0:42283:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201582.993727:0:42283:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788201583.008482:0:42283:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8289c080c8) 00000080:00200000:2.0:1788201583.008526:0:42283:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000090a1:0x1:0x0](ffff8e8289e580c8) 00000080:00200000:2.0:1788201583.008552:0:42283:0:(vvp_io.c:1826:vvp_io_init()) [0x2000090a1:0x3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1788201583.008564:0:42283:0:(vvp_io.c:304:vvp_io_fini()) [0x2000090a1:0x3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201583.008571:0:42283:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000090a1:0x3:0x0](ffff8e8289e47448) 00000080:00200000:2.0:1788201583.008599:0:42283:0:(llite_lib.c:1688:ll_put_super()) VFS Op: cfg_instance lustre-client-ffff8e82879e2000 (sb ffff8e82879e2000) 00010000:00010000:2.0:1788201583.008648:0:42283:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1a00/0x9ae3109cee9e206d lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc238e expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.008661:0:42283:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1a00/0x9ae3109cee9e206d lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1000000000000 nid: local remote: 0x7cf2114380cc238e expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.008669:0:42283:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1a00/0x9ae3109cee9e206d lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x11000400000000 nid: local remote: 0x7cf2114380cc238e expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.008681:0:42283:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1a00/0x9ae3109cee9e206d lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x11000400000000 nid: local remote: 0x7cf2114380cc238e expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.008708:0:42283:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1788201583.008715:0:42283:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8289559a00/0x9ae3109cee9e209e lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc2395 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.008722:0:42283:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8289559a00/0x9ae3109cee9e209e lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x7cf2114380cc2395 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201583.008723:0:6172:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201583.008733:0:42283:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8289559a00/0x9ae3109cee9e209e lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0x7cf2114380cc2395 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.008739:0:42283:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.121@tcp lock: ffff8e8289559a00/0x9ae3109cee9e209e lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0x7cf2114380cc2395 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.008758:0:42283:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:2.0:1788201583.008765:0:42283:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8289559c00/0x9ae3109cee9e20a5 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc239c expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.008771:0:42283:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8289559c00/0x9ae3109cee9e20a5 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x7cf2114380cc239c expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.008777:0:42283:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8289559c00/0x9ae3109cee9e20a5 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0x7cf2114380cc239c expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.008784:0:42283:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.201.121@tcp lock: ffff8e8289559c00/0x9ae3109cee9e20a5 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0x7cf2114380cc239c expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.008810:0:42283:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to regular blp list, 2 blwis in pool 10000000:01000000:2.0:1788201583.008815:0:42283:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 00010000:00010000:2.0:1788201583.008835:0:6173:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1788201583.008840:0:6173:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.121@tcp lock: ffff8e8289559a00/0x9ae3109cee9e209e lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0x7cf2114380cc2395 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.008848:0:6173:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8e8289559a00 already unused, calling callback (ffffffffc14a1c20) 10000000:00010000:2.0:1788201583.008851:0:6173:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.121@tcp lock: ffff8e8289559a00/0x9ae3109cee9e209e lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0x7cf2114380cc2395 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.008861:0:6173:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.121@tcp lock: ffff8e8289559a00/0x9ae3109cee9e209e lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0x7cf2114380cc2395 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201583.008870:0:6173:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.121@tcp lock: ffff8e8289559a00/0x9ae3109cee9e209e lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0x7cf2114380cc2395 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201583.008876:0:6173:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 10000000:01000000:2.0:1788201583.008879:0:6173:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-cliir: stopping, won't requeue 10000000:01000000:2.0:1788201583.008881:0:6173:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-cliir 00010000:00010000:3.0:1788201583.008895:0:6172:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1a00/0x9ae3109cee9e206d lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1000400000000 nid: local remote: 0x7cf2114380cc238e expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201583.008907:0:6172:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8e8290ee1a00 already unused, calling callback (ffffffffc14a1c20) 10000000:00010000:3.0:1788201583.008911:0:6172:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1a00/0x9ae3109cee9e206d lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1000400000000 nid: local remote: 0x7cf2114380cc238e expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201583.008919:0:6172:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1a00/0x9ae3109cee9e206d lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1008400000000 nid: local remote: 0x7cf2114380cc238e expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.008928:0:6173:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.121@tcp lock: ffff8e8289559a00/0x9ae3109cee9e209e lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x7cf2114380cc2395 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788201583.008929:0:6172:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1a00/0x9ae3109cee9e206d lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1009400000000 nid: local remote: 0x7cf2114380cc238e expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.008944:0:6173:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 10000000:01000000:3.0:1788201583.008946:0:6172:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:3.0:1788201583.008950:0:6172:0:(mgc_request.c:944:mgc_blocking_ast()) log lustre-client: stopping, won't requeue 10000000:01000000:3.0:1788201583.008951:0:6172:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-client 00010000:00010000:2.0:1788201583.008974:0:6173:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.121@tcp lock: ffff8e8289559a00/0x9ae3109cee9e209e lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x7cf2114380cc2395 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201583.008974:0:6172:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1a00/0x9ae3109cee9e206d lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0x7cf2114380cc238e expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201583.008992:0:6172:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201583.008993:0:6173:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.121@tcp lock: ffff8e8289559a00/0x9ae3109cee9e209e lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x7cf2114380cc2395 expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201583.009002:0:6172:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1a00/0x9ae3109cee9e206d lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0x7cf2114380cc238e expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.009008:0:6173:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788201583.009011:0:6172:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1a00/0x9ae3109cee9e206d lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0x7cf2114380cc238e expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.009014:0:6173:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.201.121@tcp lock: ffff8e8289559c00/0x9ae3109cee9e20a5 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0x7cf2114380cc239c expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.009027:0:6173:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8e8289559c00 already unused, calling callback (ffffffffc14a1c20) 00010000:00010000:3.0:1788201583.009030:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:00010000:2.0:1788201583.009031:0:6173:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.201.121@tcp lock: ffff8e8289559c00/0x9ae3109cee9e20a5 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1000400000000 nid: local remote: 0x7cf2114380cc239c expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201583.009035:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201583.009038:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201583.009066:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2404:1875062187575040:192.168.201.121@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788201583.009129:0:6173:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.121@tcp lock: ffff8e8289559c00/0x9ae3109cee9e20a5 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1008400000000 nid: local remote: 0x7cf2114380cc239c expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788201583.009143:0:6173:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.121@tcp lock: ffff8e8289559c00/0x9ae3109cee9e20a5 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x1009400000000 nid: local remote: 0x7cf2114380cc239c expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201583.009148:0:6173:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 10000000:01000000:2.0:1788201583.009150:0:6173:0:(mgc_request.c:944:mgc_blocking_ast()) log params: stopping, won't requeue 10000000:01000000:2.0:1788201583.009153:0:6173:0:(mgc_request.c:130:config_log_put()) dropping config log params 00010000:00010000:2.0:1788201583.009166:0:6173:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.201.121@tcp lock: ffff8e8289559c00/0x9ae3109cee9e20a5 lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x7cf2114380cc239c expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.009173:0:6173:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788201583.009179:0:6173:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.201.121@tcp lock: ffff8e8289559c00/0x9ae3109cee9e20a5 lrc: 1/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x7cf2114380cc239c expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.009185:0:6173:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.121@tcp lock: ffff8e8289559c00/0x9ae3109cee9e20a5 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x5809400000020 nid: local remote: 0x7cf2114380cc239c expref: -99 pid: 42118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201583.009194:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201583.009196:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201583.009198:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201583.009210:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2403:1875062187574912:192.168.201.121@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201583.009275:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2403:1875062187575168:192.168.201.121@tcp:103:ldlm_bl.0 00020000:01000000:2.0:1788201583.009893:0:42283:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0000_UUID idx: 0 gen: 1 exp: ffff8e82879e0800 active: 1 00020000:01000000:2.0:1788201583.009900:0:42283:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0001_UUID idx: 1 gen: 1 exp: ffff8e82879e1000 active: 1 00020000:01000000:2.0:1788201583.009904:0:42283:0:(lov_obd.c:64:lov_tgts_putref()) destroying 2 lov targets 00020000:01000000:2.0:1788201583.009909:0:42283:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0001_UUID : lustre-OST0001-osc-ffff8e82879e2000 00020000:01000000:2.0:1788201583.009912:0:42283:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff8e82879e2000: disconnecting target lustre-OST0001-osc-ffff8e82879e2000 00000100:00080000:2.0:1788201583.009924:0:42283:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import eaf4444c-5340-4caf-8764-d8e0f2a45321->lustre-OST0001_UUID 00000100:00080000:2.0:1788201583.010019:0:42283:0:(import.c:65:import_set_state_nolock()) ffff8e82911cf800 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:2.0:1788201583.010046:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eaf4444c-5340-4caf-8764-d8e0f2a45321:2400:1875062187575296:192.168.201.121@tcp:9:umount.0 00010000:00010000:0.0:1788201583.011684:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788201583.017525:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2403:1875062187574912:192.168.201.121@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788201583.017546:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2403:1875062187575168:192.168.201.121@tcp:103:ldlm_bl.0 00000100:00080000:2.0:1788201583.018838:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82911cf800 lustre-OST0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:2.0:1788201583.018913:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eaf4444c-5340-4caf-8764-d8e0f2a45321:2400:1875062187575296:192.168.201.121@tcp:9:umount.0 00000100:00100000:3.0:1788201583.018959:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2404:1875062187575040:192.168.201.121@tcp:103:ldlm_bl.0 00000100:00080000:0.0:1788201583.019003:0:42283:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:0.0:1788201583.019009:0:42283:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8e82879e2000: committing for last_committed 0 gen 2 00000100:00100000:0.0:1788201583.019015:0:42283:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788201583.019165:0:42283:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x9ae3109cee9e20cf 00000020:00000080:0.0:1788201583.019187:0:42283:0:(genops.c:885:class_export_put()) final put ffff8e82879e1000/LOV_OSC_UUID 00000020:01000000:0.0:1788201583.019252:0:42283:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0001-osc-ffff8e82879e2000 (flags='') 00000020:00000080:0.0:1788201583.019259:0:42283:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:0.0:1788201583.019275:0:42283:0:(obd_config.c:956:class_cleanup()) lustre-OST0001-osc-ffff8e82879e2000: forcing exports to disconnect: 0/3 00000020:00000080:3.0:1788201583.019275:0:11298:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82879e1000/LOV_OSC_UUID for lustre-OST0001-osc-ffff8e82879e2000 00000020:00080000:0.0:1788201583.019284:0:42283:0:(genops.c:1558:class_disconnect_exports()) OBD device 12 (ffff8e828961b370) has no exports 00000008:01000000:1.0:1788201583.019951:0:42283:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0001-osc-ffff8e82879e2000: client import never connected 00000100:00080000:1.0:1788201583.019957:0:42283:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:1.0:1788201583.019961:0:42283:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0001-osc-ffff8e82879e2000: committing for last_committed 0 gen 3 00000100:00100000:1.0:1788201583.019966:0:42283:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788201583.020211:0:42283:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788201583.020221:0:42283:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0001-osc-ffff8e82879e2000 (uuid eaf4444c-5340-4caf-8764-d8e0f2a45321) 00020000:01000000:1.0:1788201583.020225:0:42283:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0000_UUID : lustre-OST0000-osc-ffff8e82879e2000 00020000:01000000:1.0:1788201583.020228:0:42283:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff8e82879e2000: disconnecting target lustre-OST0000-osc-ffff8e82879e2000 00000100:00080000:1.0:1788201583.020240:0:42283:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import eaf4444c-5340-4caf-8764-d8e0f2a45321->lustre-OST0000_UUID 00000020:00000080:3.0:1788201583.020265:0:11298:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff8e82911cf800 for lustre-OST0001-osc-ffff8e82879e2000 00000020:00000080:3.0:1788201583.020277:0:11298:0:(genops.c:885:class_export_put()) final put ffff8e82911ca000/eaf4444c-5340-4caf-8764-d8e0f2a45321 00000020:00000080:3.0:1788201583.020281:0:11298:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82911ca000/eaf4444c-5340-4caf-8764-d8e0f2a45321 for lustre-OST0001-osc-ffff8e82879e2000 00000020:01000000:3.0:1788201583.020289:0:11298:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0001-osc-ffff8e82879e2000 (eaf4444c-5340-4caf-8764-d8e0f2a45321) 00000100:00080000:1.0:1788201583.024662:0:42283:0:(import.c:65:import_set_state_nolock()) ffff8e82911c9000 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788201583.025088:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eaf4444c-5340-4caf-8764-d8e0f2a45321:2400:1875062187575424:192.168.201.121@tcp:9:umount.0 00000100:00080000:3.0:1788201583.030940:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82911c9000 lustre-OST0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:3.0:1788201583.030976:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eaf4444c-5340-4caf-8764-d8e0f2a45321:2400:1875062187575424:192.168.201.121@tcp:9:umount.0 00000100:00080000:1.0:1788201583.031056:0:42283:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:1.0:1788201583.031063:0:42283:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8e82879e2000: committing for last_committed 0 gen 2 00000100:00100000:1.0:1788201583.031069:0:42283:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788201583.031210:0:42283:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x9ae3109cee9e20c8 00000020:00000080:1.0:1788201583.031232:0:42283:0:(genops.c:885:class_export_put()) final put ffff8e82879e0800/LOV_OSC_UUID 00000020:01000000:1.0:1788201583.031306:0:42283:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0000-osc-ffff8e82879e2000 (flags='') 00000020:00000080:3.0:1788201583.031502:0:11298:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82879e0800/LOV_OSC_UUID for lustre-OST0000-osc-ffff8e82879e2000 00000020:00000080:1.0:1788201583.031562:0:42283:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788201583.031574:0:42283:0:(obd_config.c:956:class_cleanup()) lustre-OST0000-osc-ffff8e82879e2000: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788201583.031580:0:42283:0:(genops.c:1558:class_disconnect_exports()) OBD device 11 (ffff8e82856fb370) has no exports 00000008:01000000:1.0:1788201583.031846:0:42283:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0000-osc-ffff8e82879e2000: client import never connected 00000100:00080000:1.0:1788201583.031850:0:42283:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:1.0:1788201583.031853:0:42283:0:(client.c:3125:ptlrpc_free_committed()) lustre-OST0000-osc-ffff8e82879e2000: committing for last_committed 0 gen 3 00000100:00100000:1.0:1788201583.031856:0:42283:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788201583.032585:0:42283:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788201583.032597:0:42283:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0000-osc-ffff8e82879e2000 (uuid eaf4444c-5340-4caf-8764-d8e0f2a45321) 00000020:00000080:1.0:1788201583.032602:0:42283:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x9ae3109cee9e20c1 00000020:00000080:1.0:1788201583.032610:0:42283:0:(genops.c:885:class_export_put()) final put ffff8e82879e3000/eaf4444c-5340-4caf-8764-d8e0f2a45321 00000020:00000080:0.0:1788201583.033794:0:11298:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff8e82911c9000 for lustre-OST0000-osc-ffff8e82879e2000 00000020:00000080:0.0:1788201583.033809:0:11298:0:(genops.c:885:class_export_put()) final put ffff8e82911ca800/eaf4444c-5340-4caf-8764-d8e0f2a45321 00000020:00000080:0.0:1788201583.033812:0:11298:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82911ca800/eaf4444c-5340-4caf-8764-d8e0f2a45321 for lustre-OST0000-osc-ffff8e82879e2000 00000020:01000000:0.0:1788201583.033827:0:11298:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0000-osc-ffff8e82879e2000 (eaf4444c-5340-4caf-8764-d8e0f2a45321) 00000100:00080000:1.0:1788201583.034495:0:42283:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import eaf4444c-5340-4caf-8764-d8e0f2a45321->lustre-MDT0000_UUID 00010000:00010000:1.0:1788201583.034553:0:42283:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289561e00/0x9ae3109cee9e2115 lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc24de expref: -99 pid: 42252 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201583.034620:0:42283:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289561e00/0x9ae3109cee9e2115 lrc: 1/0,0 mode: --/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc24de expref: -99 pid: 42252 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201583.034628:0:42283:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788201583.034649:0:42283:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f5480 going to sleep for 0 seconds 00000100:00100000:1.0:1788201583.034662:0:42283:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job umount:eaf4444c-5340-4caf-8764-d8e0f2a45321:42283:1875062187575552:192.168.201.121@tcp:103:umount.0 00010000:00010000:1.0:1788201583.034674:0:42283:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788201583.034684:0:42283:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788201583.034689:0:42283:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e8289561e00/0x9ae3109cee9e2115 lrc: 0/0,0 mode: --/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc24de expref: -99 pid: 42252 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201583.034784:0:42283:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201583.034807:0:42283:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788201583.034813:0:42283:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:1.0:1788201583.034819:0:42283:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f5480 going to sleep for 0 seconds 00000100:00100000:1.0:1788201583.034824:0:42283:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job umount:eaf4444c-5340-4caf-8764-d8e0f2a45321:42283:1875062187575680:192.168.201.121@tcp:103:umount.0 00010000:00010000:1.0:1788201583.034829:0:42283:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:1.0:1788201583.034833:0:42283:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:1.0:1788201583.034837:0:42283:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e82879e2000 lock: ffff8e828b94b600/0x9ae3109cee9e23ed lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc2ed2 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00080000:1.0:1788201583.036706:0:42283:0:(import.c:65:import_set_state_nolock()) ffff8e82911ce000 lustre-MDT0000_UUID: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788201583.036779:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eaf4444c-5340-4caf-8764-d8e0f2a45321:2400:1875062187575808:192.168.201.121@tcp:39:umount.0 00000020:00000080:2.0:1788201583.040232:0:37:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82879e3000/eaf4444c-5340-4caf-8764-d8e0f2a45321 for lustre-clilov-ffff8e82879e2000 00000100:00080000:3.0:1788201583.042925:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82911ce000 lustre-MDT0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:3.0:1788201583.042994:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eaf4444c-5340-4caf-8764-d8e0f2a45321:2400:1875062187575808:192.168.201.121@tcp:39:umount.0 00000100:00080000:0.0:1788201583.043708:0:42283:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:0.0:1788201583.043713:0:42283:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e82879e2000: committing for last_committed 81604378725 gen 2 00000100:00100000:0.0:1788201583.043727:0:42283:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8e828d715500 x1875062187538816/t81604378728(81604378728) o36->lustre-MDT0000-mdc-ffff8e82879e2000@192.168.201.121@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201589 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201583.043753:0:42283:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8e828793dc00 x1875062187539456/t81604378729(81604378729) o36->lustre-MDT0000-mdc-ffff8e82879e2000@192.168.201.121@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201589 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201583.043768:0:42283:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8e828d6a5880 x1875062187541120/t81604378730(81604378730) o36->lustre-MDT0000-mdc-ffff8e82879e2000@192.168.201.121@tcp:12/10 lens 512/448 e 0 to 0 dl 1788201595 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201583.043783:0:42283:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8e828d722300 x1875062187575808/t81604378831(81604378831) o39->lustre-MDT0000-mdc-ffff8e82879e2000@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201593 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201583.043800:0:42283:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788201583.046965:0:42283:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x9ae3109cee9e20b3 00000020:00000080:0.0:1788201583.046987:0:42283:0:(genops.c:885:class_export_put()) final put ffff8e82911cb000/eaf4444c-5340-4caf-8764-d8e0f2a45321 00000100:00080000:0.0:1788201583.047814:0:42283:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import eaf4444c-5340-4caf-8764-d8e0f2a45321->lustre-MDT0001_UUID 00000020:00000080:2.0:1788201583.051486:0:37:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82911cb000/eaf4444c-5340-4caf-8764-d8e0f2a45321 for lustre-MDT0000-mdc-ffff8e82879e2000 00000100:00080000:0.0:1788201583.052546:0:42283:0:(import.c:65:import_set_state_nolock()) ffff8e82911c8000 lustre-MDT0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:1.0:1788201583.052720:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eaf4444c-5340-4caf-8764-d8e0f2a45321:2400:1875062187575936:192.168.201.121@tcp:39:umount.0 00000100:00080000:1.0:1788201583.059603:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82911c8000 lustre-MDT0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:1.0:1788201583.059672:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eaf4444c-5340-4caf-8764-d8e0f2a45321:2400:1875062187575936:192.168.201.121@tcp:39:umount.0 00000100:00080000:0.0:1788201583.059720:0:42283:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:0.0:1788201583.059725:0:42283:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8e82879e2000: committing for last_committed 0 gen 2 00000100:00100000:0.0:1788201583.059736:0:42283:0:(client.c:3163:ptlrpc_free_committed()) @@@ free request with old gen req@ffff8e828793dc00 x1875062187575936/t4294967341(4294967341) o39->lustre-MDT0001-mdc-ffff8e82879e2000@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201593 ref 1 fl Complete:RNQU/204/0 rc 0/0 job:'umount.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201583.059947:0:42283:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:0.0:1788201583.064531:0:42283:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x9ae3109cee9e20ba 00000020:00000080:0.0:1788201583.064553:0:42283:0:(genops.c:885:class_export_put()) final put ffff8e82911cf000/eaf4444c-5340-4caf-8764-d8e0f2a45321 00000020:00000080:1.0:1788201583.064753:0:42283:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x9ae3109cee9e20ac 00000020:00000080:1.0:1788201583.064759:0:42283:0:(genops.c:885:class_export_put()) final put ffff8e82911cd800/eaf4444c-5340-4caf-8764-d8e0f2a45321 00000020:00000080:3.0:1788201583.064842:0:37:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82911cf000/eaf4444c-5340-4caf-8764-d8e0f2a45321 for lustre-MDT0001-mdc-ffff8e82879e2000 00000020:01000000:1.0:1788201583.064991:0:42283:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff8e82879e2000 (flags='') 00000020:00000080:1.0:1788201583.065005:0:42283:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788201583.065013:0:42283:0:(obd_config.c:956:class_cleanup()) lustre-clilov-ffff8e82879e2000: forcing exports to disconnect: 0/1 00000020:00080000:1.0:1788201583.065019:0:42283:0:(genops.c:1558:class_disconnect_exports()) OBD device 7 (ffff8e8291bd99b8) has no exports 00000020:00000080:1.0:1788201583.065111:0:42283:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788201583.065118:0:42283:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff8e82879e2000 (uuid eaf4444c-5340-4caf-8764-d8e0f2a45321) 00000020:00000080:1.0:1788201583.065122:0:42283:0:(genops.c:885:class_export_put()) final put ffff8e82911c8800/eaf4444c-5340-4caf-8764-d8e0f2a45321 00000020:00000080:1.0:1788201583.065124:0:42283:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82911c8800/eaf4444c-5340-4caf-8764-d8e0f2a45321 for lustre-clilov-ffff8e82879e2000 00000020:01000000:1.0:1788201583.065133:0:42283:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff8e82879e2000 (eaf4444c-5340-4caf-8764-d8e0f2a45321) 00000020:01000000:1.0:1788201583.065364:0:42283:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff8e82879e2000 (flags='') 00000020:00000080:1.0:1788201583.065369:0:42283:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788201583.065373:0:42283:0:(obd_config.c:956:class_cleanup()) lustre-clilmv-ffff8e82879e2000: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788201583.065375:0:42283:0:(genops.c:1558:class_disconnect_exports()) OBD device 8 (ffff8e82856fcd28) has no exports 00000020:00000080:1.0:1788201583.065433:0:42283:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788201583.065439:0:42283:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff8e82879e2000 (uuid eaf4444c-5340-4caf-8764-d8e0f2a45321) 00000020:01000000:1.0:1788201583.065442:0:42283:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff8e82879e2000 (flags='') 00000020:00000080:1.0:1788201583.065445:0:42283:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788201583.065449:0:42283:0:(obd_config.c:956:class_cleanup()) lustre-MDT0000-mdc-ffff8e82879e2000: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788201583.065451:0:42283:0:(genops.c:1558:class_disconnect_exports()) OBD device 9 (ffff8e82856f8000) has no exports 00000020:00000080:3.0:1788201583.066229:0:37:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82911cd800/eaf4444c-5340-4caf-8764-d8e0f2a45321 for lustre-clilmv-ffff8e82879e2000 00000020:00000080:3.0:1788201583.066237:0:37:0:(genops.c:885:class_export_put()) final put ffff8e82911cc000/eaf4444c-5340-4caf-8764-d8e0f2a45321 00000020:00000080:3.0:1788201583.066286:0:37:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82911cc000/eaf4444c-5340-4caf-8764-d8e0f2a45321 for lustre-clilmv-ffff8e82879e2000 00000020:01000000:3.0:1788201583.066294:0:37:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilmv-ffff8e82879e2000 (eaf4444c-5340-4caf-8764-d8e0f2a45321) 00000008:01000000:1.0:1788201583.066901:0:42283:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff8e82879e2000: client import never connected 00000100:00080000:1.0:1788201583.066909:0:42283:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:1.0:1788201583.066914:0:42283:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e82879e2000: committing for last_committed 81604378725 gen 3 00000100:00100000:1.0:1788201583.066924:0:42283:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:1.0:1788201583.071153:0:42283:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788201583.071175:0:42283:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0000-mdc-ffff8e82879e2000 (uuid eaf4444c-5340-4caf-8764-d8e0f2a45321) 00000020:01000000:1.0:1788201583.071182:0:42283:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0001-mdc-ffff8e82879e2000 (flags='') 00000020:00000080:1.0:1788201583.071193:0:42283:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788201583.071198:0:42283:0:(obd_config.c:956:class_cleanup()) lustre-MDT0001-mdc-ffff8e82879e2000: forcing exports to disconnect: 0/2 00000020:00080000:1.0:1788201583.071205:0:42283:0:(genops.c:1558:class_disconnect_exports()) OBD device 10 (ffff8e82856f99b8) has no exports 00000020:00000080:3.0:1788201583.072040:0:37:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff8e82911ce000 for lustre-MDT0000-mdc-ffff8e82879e2000 00000020:00000080:3.0:1788201583.072054:0:37:0:(genops.c:885:class_export_put()) final put ffff8e82911cb800/eaf4444c-5340-4caf-8764-d8e0f2a45321 00000020:00000080:3.0:1788201583.072059:0:37:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82911cb800/eaf4444c-5340-4caf-8764-d8e0f2a45321 for lustre-MDT0000-mdc-ffff8e82879e2000 00000020:01000000:3.0:1788201583.072069:0:37:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0000-mdc-ffff8e82879e2000 (eaf4444c-5340-4caf-8764-d8e0f2a45321) 00000008:01000000:1.0:1788201583.073564:0:42283:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0001-mdc-ffff8e82879e2000: client import never connected 00000100:00080000:1.0:1788201583.073574:0:42283:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:1.0:1788201583.073579:0:42283:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff8e82879e2000: committing for last_committed 0 gen 3 00000100:00100000:1.0:1788201583.073588:0:42283:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:3.0:1788201583.077558:0:37:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff8e82911c8000 for lustre-MDT0001-mdc-ffff8e82879e2000 00000020:00000080:1.0:1788201583.077671:0:42283:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788201583.077688:0:42283:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0001-mdc-ffff8e82879e2000 (uuid eaf4444c-5340-4caf-8764-d8e0f2a45321) 00000020:00000080:1.0:1788201583.077694:0:42283:0:(genops.c:885:class_export_put()) final put ffff8e82911c9800/eaf4444c-5340-4caf-8764-d8e0f2a45321 00000020:00000080:1.0:1788201583.077698:0:42283:0:(genops.c:844:class_export_destroy()) destroying export ffff8e82911c9800/eaf4444c-5340-4caf-8764-d8e0f2a45321 for lustre-MDT0001-mdc-ffff8e82879e2000 00000020:01000000:1.0:1788201583.077707:0:42283:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0001-mdc-ffff8e82879e2000 (eaf4444c-5340-4caf-8764-d8e0f2a45321) 00000080:02000400:0.0:1788201583.079622:0:42283:0:(llite_lib.c:1739:ll_put_super()) Unmounted lustre-client 00000020:01000000:0.0:1788201583.079631:0:42283:0:(obd_config.c:1198:class_del_profile()) Del profile lustre-client 00000020:01000004:0.0:1788201583.079691:0:42283:0:(obd_mount.c:897:lustre_common_put_super()) dropping sb ffff8e82879e2000 00000020:01000004:0.0:1788201583.079698:0:42283:0:(obd_mount.c:578:lustre_stop_mgc()) MGC still has 1 references. 00000020:01000004:0.0:1788201583.079700:0:42283:0:(obd_mount.c:910:lustre_common_put_super()) MGC still in use 00000020:01000004:0.0:1788201583.079702:0:42283:0:(obd_mount.c:710:lustre_put_lsi()) put ffff8e82879e2000 1 00000020:01000004:0.0:1788201583.079705:0:42283:0:(obd_mount.c:668:lustre_free_lsi()) Freeing lsi ffff8e82879e5000 00010000:00010000:3.0:1788201583.813939:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201584.836489:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201585.862486:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201586.184054:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187576192:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201586.184197:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187576448:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201586.184234:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:0.0:1788201586.184240:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187576064:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201586.184267:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187576320:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201586.184287:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187576576:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201586.192699:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187576320:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201586.192729:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187576576:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201586.192742:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187576448:192.168.201.121@tcp:400:kworker.0 00000100:00080000:1.0:1788201586.193087:0:2401:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff8e8282a4b800 of lustre-MDT0000_UUID@192.168.201.121@tcp abruptly disconnected: reconnecting 00000100:00080000:1.0:1788201586.193094:0:2401:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:1.0:1788201586.193099:0:2401:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8e8282a4b800: Connection to lustre-MDT0000 (at 192.168.201.121@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:1.0:1788201586.193106:0:2401:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201586.193111:0:2401:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 2950 00000100:00080000:1.0:1788201586.193122:0:2401:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201586.193136:0:2401:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201586.193171:0:2401:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187576704/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201586.193272:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187576192:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201586.193345:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187576704:192.168.201.121@tcp:38:ptlrpcd_00_00.0 00000100:00080000:3.0:1788201586.198617:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201586.198645:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-19) 00000100:00100000:3.0:1788201586.198743:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187576704:192.168.201.121@tcp:38:ptlrpcd_00_00.0 00010000:00010000:1.0:1788201586.884494:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201587.908625:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201588.932581:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201589.963059:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201590.980489:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201591.302746:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201591.302755:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201591.302760:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201591.302773:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3046 00000100:00080000:2.0:1788201591.302788:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1788201591.302797:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201591.302811:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201591.302833:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894d4700 x1875062187576960/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201591.302919:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187576960:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201591.304051:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2404:1875062187576832:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201591.305913:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187577088:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201591.306450:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187577216:192.168.201.121@tcp:400:kworker.0 00000100:00080000:2.0:1788201591.324832:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201591.324854:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-19) 00000100:00100000:2.0:1788201591.324875:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187576960:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201591.328095:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187577344:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201591.332598:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187577216:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201591.333381:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187577344:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201591.335416:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187577088:192.168.201.121@tcp:400:kworker.0 00010000:00010000:0.0:1788201592.004892:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201593.030341:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201594.053408:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201595.083451:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201596.100558:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201596.421025:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201596.421034:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201596.421037:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201596.421047:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3051 00000100:00080000:1.0:1788201596.421057:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1788201596.421064:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201596.421084:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201596.421102:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187577600/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201596.421172:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187577728:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201596.421338:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187577984:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201596.421373:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788201596.421381:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187577472:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201596.421407:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187577856:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201596.421935:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187577600:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201596.425215:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187577856:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201596.425252:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187577984:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201596.426179:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187577728:192.168.201.121@tcp:400:kworker.0 00000100:00080000:2.0:1788201596.426379:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201596.426387:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-19) 00000100:00100000:2.0:1788201596.426391:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187577600:192.168.201.121@tcp:38:kworker.0 00010000:00010000:2.0:1788201597.125014:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201598.151335:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201599.178703:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201600.196514:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201601.220944:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201601.543964:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201601.543975:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201601.543980:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201601.544114:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3056 00000100:00080000:0.0:1788201601.544130:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1788201601.544140:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201601.544153:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201601.544175:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187578240/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201601.544261:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187578112:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201601.544859:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187578496:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201601.545523:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187578240:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201601.545758:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:0.0:1788201601.545768:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187578368:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201601.545803:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187578624:192.168.201.121@tcp:400:kworker.0 00000100:00080000:1.0:1788201601.559647:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201601.559669:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-19) 00000100:00100000:1.0:1788201601.559676:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187578240:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788201601.559816:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187578368:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201601.559832:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187578624:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201601.559844:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187578496:192.168.201.121@tcp:400:kworker.0 00010000:00010000:1.0:1788201602.244606:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1788201602.564423:0:2401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201586/real 1788201586] req@ffff8e828793dc00 x1875062187576064/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201602 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201602.564459:0:2401:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8e828793dc00 x1875062187576064/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201602 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201602.564468:0:2401:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from FULL to DISCONN 00000100:02020000:1.0:1788201602.564472:0:2401:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail 00000100:00080000:1.0:1788201602.564475:0:2401:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.201.121@tcp for MGC192.168.201.121@tcp not replayable, auto-deactivating 00000100:00080000:1.0:1788201602.564478:0:2401:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:1.0:1788201602.564481:0:2401:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8e828793dc00 x1875062187576064/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201602 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201602.564490:0:2401:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8e82894d7800 x1875062187576832/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201607 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201602.564517:0:2401:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8e828d722a00 x1875062187577472/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201612 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201602.564539:0:2401:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8e828793ea00 x1875062187578112/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201617 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:1.0:1788201602.564548:0:2401:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:1.0:1788201602.564551:0:2401:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:1.0:1788201602.564552:0:2401:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:1.0:1788201602.564554:0:2401:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:1.0:1788201602.564585:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187578112:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201602.564591:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187576064:192.168.201.121@tcp:400:kworker.0 00000100:00080000:1.0:1788201602.564631:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201602.564636:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201602.564638:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201602.564644:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 2949 00000100:00080000:1.0:1788201602.564655:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788201602.564683:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721f80 x1875062187578752/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201602.564699:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201602.564703:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201602.564704:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201602.564708:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3061 00000100:00080000:1.0:1788201602.564711:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201602.564714:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201602.564723:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201602.564732:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723b80 x1875062187578880/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201602.564774:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187578752:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788201602.564842:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187578880:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788201602.564874:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187579008:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201602.564908:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187579264:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201602.565072:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2404:1875062187576832:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201602.565193:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187579136:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201602.569370:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187577472:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201602.578379:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187579136:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201602.578428:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187579008:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201602.578439:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187579264:192.168.201.121@tcp:400:kworker.0 00000100:00080000:2.0:1788201602.583450:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201602.583464:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-19) 00000100:00100000:2.0:1788201602.583470:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187578880:192.168.201.121@tcp:38:kworker.0 00010000:00010000:2.0:1788201603.270635:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201604.292776:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201605.316669:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201606.341925:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201607.365536:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201607.684454:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201607.684467:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201607.684471:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201607.684475:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201607.684480:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201607.684494:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3062 00000100:00080000:0.0:1788201607.684519:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201607.684528:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201607.684543:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201607.684611:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187579392/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201607.684840:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187579520:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201607.685154:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187579776:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201607.685200:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788201607.685207:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187579648:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201607.685591:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187579392:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201607.690798:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187579648:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201607.690832:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187579520:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201607.690846:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187579776:192.168.201.121@tcp:400:kworker.0 00000100:00080000:2.0:1788201607.691978:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201607.692004:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-19) 00000100:00100000:2.0:1788201607.692011:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187579392:192.168.201.121@tcp:38:kworker.0 00010000:00010000:1.0:1788201608.390463:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201609.414008:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201610.436661:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201611.468212:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201612.484617:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201612.804578:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201612.804589:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201612.804592:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201612.804596:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201612.804600:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201612.804610:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3067 00000100:00080000:1.0:1788201612.804621:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201612.804628:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201612.804638:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201612.804672:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187579904/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201612.804838:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201602/real 1788201602] req@ffff8e828d721f80 x1875062187578752/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201612 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201612.804864:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721f80 x1875062187578752/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201612 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201612.804884:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187579904:192.168.201.121@tcp:38:kworker.0 00000100:00080000:2.0:1788201612.804996:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201612.805021:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201612.805025:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187578752:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788201612.805187:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187580032:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201612.805221:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187580288:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201612.805248:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788201612.805255:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187580160:192.168.201.121@tcp:400:kworker.0 00000100:00080000:3.0:1788201612.807263:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201612.807275:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201612.807278:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201612.807287:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3062 00000100:00080000:3.0:1788201612.807340:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788201612.807347:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788201612.807382:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187580416/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201612.808495:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187580416:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788201612.809164:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201612.809171:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-19) 00000100:00100000:2.0:1788201612.809176:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187579904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201612.809294:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187580032:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201612.812610:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187580160:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201612.812628:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187580288:192.168.201.121@tcp:400:kworker.0 00000100:00080000:2.0:1788201612.824149:0:2400:0:(import.c:1132:ptlrpc_connect_interpret()) MGC192.168.201.121@tcp: connect to target with instance 0 10000000:01000000:2.0:1788201612.824165:0:2400:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:02000400:2.0:1788201612.824174:0:2400:0:(import.c:1284:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.201.121@tcp) after server handle changed from 0x7cf2114380cbf7b9 to 0x7cf2114380cc2ef5 00000100:00080000:2.0:1788201612.824180:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:2.0:1788201612.824185:0:2400:0:(import.c:1674:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.201.121@tcp; invalidating 00000100:00100000:2.0:1788201612.824531:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187580416:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788201612.824582:0:42506:0:(import.c:1598:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.201.121@tcp to MGS@192.168.201.121@tcp 00000100:00100000:0.0:1788201612.824586:0:42506:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:0.0:1788201612.824588:0:42506:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:0.0:1788201612.824601:0:42506:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.121@tcp lock: ffff8e8287992000/0x9ae3109cee9e1ad2 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0x7cf2114380cbf88b expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201612.824609:0:42506:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.121@tcp lock: ffff8e8287992000/0x9ae3109cee9e1ad2 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0x7cf2114380cbf88b expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201612.824613:0:42506:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201612.824626:0:42506:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201612.824635:0:42506:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.121@tcp lock: ffff8e8287992000/0x9ae3109cee9e1ad2 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0x7cf2114380cbf88b expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201612.824646:0:42506:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.121@tcp lock: ffff8e8287992200/0x9ae3109cee9e1ae0 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x7cf2114380cbf899 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201612.824650:0:42506:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.121@tcp lock: ffff8e8287992200/0x9ae3109cee9e1ae0 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x7cf2114380cbf899 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201612.824654:0:42506:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201612.824656:0:42506:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201612.824658:0:42506:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.121@tcp lock: ffff8e8287992200/0x9ae3109cee9e1ae0 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x7cf2114380cbf899 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201612.824668:0:42506:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1400/0x9ae3109cee9e1acb lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x7cf2114380cbf884 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201612.824672:0:42506:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1400/0x9ae3109cee9e1acb lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x7cf2114380cbf884 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201612.824674:0:42506:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:0.0:1788201612.824677:0:42506:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201612.824680:0:42506:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1400/0x9ae3109cee9e1acb lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x7cf2114380cbf884 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201612.824691:0:42506:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1e00/0x9ae3109cee9e1ad9 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x7cf2114380cbf892 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:0.0:1788201612.824697:0:42506:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1e00/0x9ae3109cee9e1ad9 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x7cf2114380cbf892 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:0.0:1788201612.824702:0:42506:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:0.0:1788201612.824705:0:42506:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788201612.824709:0:42506:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.201.121@tcp lock: ffff8e8290ee1e00/0x9ae3109cee9e1ad9 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x7cf2114380cbf892 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:0.0:1788201612.824721:0:42506:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:0.0:1788201612.824726:0:42506:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from RECOVER to FULL 10000000:01000000:0.0:1788201612.824728:0:42506:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:0.0:1788201612.824731:0:42506:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:0.0:1788201612.824736:0:42506:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) MGC192.168.201.121@tcp: Connection restored to 192.168.201.121@tcp (at 192.168.201.121@tcp) 00010000:00010000:2.0:1788201613.515448:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788201614.021554:0:6175:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788201614.021563:0:6175:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff8e8282a4b800 from 79 10000000:01000000:2.0:1788201614.021571:0:6175:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788201614.021635:0:6175:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8289463e00/0x9ae3109cee9e23f4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201614.021647:0:6175:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.121@tcp lock: ffff8e8289463e00/0x9ae3109cee9e23f4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201614.021659:0:6175:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.121@tcp lock: ffff8e8289463e00/0x9ae3109cee9e23f4 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201614.021687:0:6175:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187580544:192.168.201.121@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788201614.021796:0:6175:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230180 going to sleep for 11 seconds 00000100:00100000:2.0:1788201614.026404:0:6175:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187580544:192.168.201.121@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788201614.026437:0:6175:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.121@tcp lock: ffff8e8289463e00/0x9ae3109cee9e23f4 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc3022 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201614.026450:0:6175:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.121@tcp lock: ffff8e8289463e00/0x9ae3109cee9e23f4 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x7cf2114380cc3022 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201614.026535:0:6175:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187580672:192.168.201.121@tcp:256:ll_cfg_requeue.0 00000100:00100000:2.0:1788201614.026625:0:6175:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230180 going to sleep for 11 seconds 00000100:00100000:2.0:1788201614.039553:0:6175:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187580672:192.168.201.121@tcp:256:ll_cfg_requeue.0 00000100:00080000:2.0:1788201614.039619:0:6175:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8e82871d2800 already not connected (conn 152, was 0): DISCONN 00010000:00080000:2.0:1788201614.039631:0:6175:0:(ldlm_lib.c:85:import_set_conn()) imp ffff8e82871d2800@lustre-MDT0000-mdc-ffff8e8282a4b800: found existing conn 192.168.201.121@tcp, moved to head 00000100:00080000:2.0:1788201614.039636:0:6175:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201614.039641:0:6175:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 0 00000100:00080000:2.0:1788201614.039650:0:6175:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201614.039670:0:6175:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201614.039698:0:6175:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894d4700 x1875062187580800/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 10000000:01000000:2.0:1788201614.039753:0:6175:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.121@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:2.0:1788201614.039762:0:6175:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8289463e00/0x9ae3109cee9e23f4 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc3022 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201614.039773:0:6175:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.121@tcp lock: ffff8e8289463e00/0x9ae3109cee9e23f4 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc3022 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201614.039782:0:6175:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:2.0:1788201614.039785:0:6175:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff8e8282a4b800 from 52 10000000:01000000:2.0:1788201614.039788:0:6175:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:2.0:1788201614.039813:0:6175:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8289462e00/0x9ae3109cee9e23fb lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201614.039821:0:6175:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.121@tcp lock: ffff8e8289462e00/0x9ae3109cee9e23fb lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201614.039831:0:6175:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.121@tcp lock: ffff8e8289462e00/0x9ae3109cee9e23fb lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201614.039849:0:6175:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187580928:192.168.201.121@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788201614.039922:0:6175:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230180 going to sleep for 11 seconds 00000100:00100000:1.0:1788201614.040029:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187580800:192.168.201.121@tcp:38:ll_cfg_requeue.0 00000100:00100000:1.0:1788201614.050805:0:6175:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7100 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187580928:192.168.201.121@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201614.050834:0:6175:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.121@tcp lock: ffff8e8289462e00/0x9ae3109cee9e23fb lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc3029 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201614.050847:0:6175:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.121@tcp lock: ffff8e8289462e00/0x9ae3109cee9e23fb lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x7cf2114380cc3029 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201614.050937:0:6175:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187581056:192.168.201.121@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201614.051018:0:6175:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f5cc0 going to sleep for 11 seconds 00010000:00080000:2.0:1788201614.051475:0:2400:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff8e82894d4700 x1875062187580800/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/416 e 0 to 0 dl 1788201624 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201614.051493:0:2400:0:(import.c:1132:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to target with instance 20 00000100:00080000:2.0:1788201614.051511:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:2.0:1788201614.051541:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201614.051544:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201614.051551:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201589 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201614.051558:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 0 00000100:00080000:2.0:1788201614.051564:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 0 to 0 dl 1788201589 ref 1 fl New:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201614.051574:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187580800:192.168.201.121@tcp:38:ll_cfg_requeue.0 00000100:00100000:2.0:1788201614.051591:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187537920:192.168.201.121@tcp:36:mcreate.0 00000100:00100000:1.0:1788201614.060923:0:6175:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187581056:192.168.201.121@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201614.060987:0:6175:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187581184:192.168.201.121@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788201614.061090:0:6175:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f5cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201614.066744:0:6175:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187581184:192.168.201.121@tcp:503:ll_cfg_requeue.0 00000040:00080000:1.0:1788201614.067785:0:42516:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 51 00000020:01000000:2.0:1788201614.068034:0:6175:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 52-51 (rc=0) 10000000:01000000:2.0:1788201614.068053:0:6175:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.121@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:2.0:1788201614.068063:0:6175:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8289462e00/0x9ae3109cee9e23fb lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc3029 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201614.068074:0:6175:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.121@tcp lock: ffff8e8289462e00/0x9ae3109cee9e23fb lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc3029 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788201614.068084:0:6175:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:2.0:1788201614.068086:0:6175:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff8e8282a4b800 from 7 10000000:01000000:2.0:1788201614.068089:0:6175:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1788201614.068124:0:6175:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8289463c00/0x9ae3109cee9e2402 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201614.068133:0:6175:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.121@tcp lock: ffff8e8289463c00/0x9ae3109cee9e2402 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201614.068148:0:6175:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.121@tcp lock: ffff8e8289463c00/0x9ae3109cee9e2402 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201614.068182:0:6175:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4700 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187581312:192.168.201.121@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788201614.068253:0:6175:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82922306c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201614.070397:0:6175:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4700 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187581312:192.168.201.121@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788201614.070426:0:6175:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.121@tcp lock: ffff8e8289463c00/0x9ae3109cee9e2402 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc3030 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201614.070438:0:6175:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.121@tcp lock: ffff8e8289463c00/0x9ae3109cee9e2402 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x7cf2114380cc3030 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201614.070512:0:6175:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187581440:192.168.201.121@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201614.070578:0:6175:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f5540 going to sleep for 11 seconds 00000100:00100000:1.0:1788201614.082477:0:6175:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187581440:192.168.201.121@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788201614.082537:0:6175:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187581568:192.168.201.121@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788201614.082618:0:6175:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82856f5540 going to sleep for 11 seconds 00000100:00100000:2.0:1788201614.085955:0:6175:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187581568:192.168.201.121@tcp:503:ll_cfg_requeue.0 00000040:00080000:2.0:1788201614.086913:0:42517:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000020:01000000:3.0:1788201614.087115:0:6175:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:3.0:1788201614.087135:0:6175:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.121@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:3.0:1788201614.087145:0:6175:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e8289463c00/0x9ae3109cee9e2402 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc3030 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201614.087156:0:6175:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.121@tcp lock: ffff8e8289463c00/0x9ae3109cee9e2402 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc3030 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788201614.087175:0:6175:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:3.0:1788201614.087178:0:6175:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff8e82871a8000 from 1 10000000:01000000:3.0:1788201614.087181:0:6175:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:3.0:1788201614.087214:0:6175:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e828b94a400/0x9ae3109cee9e2409 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201614.087223:0:6175:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.201.121@tcp lock: ffff8e828b94a400/0x9ae3109cee9e2409 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201614.087232:0:6175:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.201.121@tcp lock: ffff8e828b94a400/0x9ae3109cee9e2409 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201614.087255:0:6175:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187581696:192.168.201.121@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1788201614.087370:0:6175:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788201614.091848:0:6175:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187581696:192.168.201.121@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788201614.091889:0:6175:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.201.121@tcp lock: ffff8e828b94a400/0x9ae3109cee9e2409 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc3037 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201614.091902:0:6175:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.201.121@tcp lock: ffff8e828b94a400/0x9ae3109cee9e2409 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x7cf2114380cc3037 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201614.091984:0:6175:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187581824:192.168.201.121@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1788201614.092112:0:6175:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788201614.113055:0:6175:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:6175:1875062187581824:192.168.201.121@tcp:501:ll_cfg_requeue.0 10000000:01000000:3.0:1788201614.113102:0:6175:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.201.121@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:3.0:1788201614.113111:0:6175:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.201.121@tcp lock: ffff8e828b94a400/0x9ae3109cee9e2409 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc3037 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201614.113121:0:6175:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.201.121@tcp lock: ffff8e828b94a400/0x9ae3109cee9e2409 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x7cf2114380cc3037 expref: -99 pid: 6175 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201614.535292:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201615.557576:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201616.581259:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201617.604636:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201617.924512:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 1 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201617.924518:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:3.0:1788201617.924560:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187582080:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201617.924561:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2404:1875062187581952:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201617.925004:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187582208:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201617.933089:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187582336:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201617.936467:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187582080:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201617.940198:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2404:1875062187581952:192.168.201.121@tcp:400:kworker.0 00010000:00010000:2.0:1788201617.947013:0:2404:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 1 blwis in pool 00000100:00100000:2.0:1788201617.947033:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187582208:192.168.201.121@tcp:400:kworker.0 00010000:00010000:3.0:1788201617.948243:0:2403:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 2 blwis in pool 00000100:00100000:3.0:1788201617.948257:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187582336:192.168.201.121@tcp:400:kworker.0 00010000:00010000:2.0:1788201617.948343:0:25294:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (1 blwis in pool) 00010000:00010000:2.0:1788201617.948358:0:25294:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788201617.948364:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201617.948366:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201617.948367:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201617.948443:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201618.628776:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201618.948489:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201618.948496:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:3.0:1788201618.948544:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187582592:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201618.948644:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187582848:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201618.948683:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788201618.948689:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2403:1875062187582464:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201618.948876:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187582720:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201618.952173:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2403:1875062187582464:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201618.952193:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187582720:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201618.952204:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187582592:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201618.952214:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187582848:192.168.201.121@tcp:400:kworker.0 00010000:00010000:2.0:1788201619.659587:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201620.678261:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201621.702191:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788201622.726492:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201623.750348:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1788201623.892057:0:42607:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8e8292531988), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788201623.892071:0:42607:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201623.892097:0:42607:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201623.892117:0:42607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201623.892134:0:42607:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201623.892142:0:42607:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201623.892150:0:42607:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201623.892153:0:42607:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e23e6 00000080:00010000:1.0:1788201623.892161:0:42607:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23e6 from it ffffaa1800c97cb8 00010000:00010000:1.0:1788201623.892165:0:42607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201623.892173:0:42607:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200008:1.0:1788201623.893024:0:42607:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff8e8292531988), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788201623.893030:0:42607:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201623.893042:0:42607:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201623.893052:0:42607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201623.893061:0:42607:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201623.893067:0:42607:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201623.893071:0:42607:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201623.893074:0:42607:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e23e6 00000080:00010000:1.0:1788201623.893079:0:42607:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23e6 from it ffffaa1800c97cb8 00010000:00010000:1.0:1788201623.893082:0:42607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201623.893090:0:42607:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:1.0:1788201623.893182:0:42607:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201623.893190:0:42607:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201623.893197:0:42607:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788201623.893206:0:42607:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:1.0:1788201623.893212:0:42607:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201623.893216:0:42607:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201623.893218:0:42607:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e23e6 00000080:00010000:1.0:1788201623.893222:0:42607:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23e6 from it ffffaa1800c97bb0 00010000:00010000:1.0:1788201623.893225:0:42607:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788201623.893232:0:42607:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:1.0:1788201623.893239:0:42607:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 44 00000080:00200008:1.0:1788201623.893246:0:42607:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff8e8292531988), flags 100000 00000080:00200080:1.0:1788201623.893341:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:1.0:1788201623.893357:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8291bb9400 uarg=0000000000000000 00800000:00000004:1.0:1788201623.893365:0:42607:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff8e8282a4b800: age 3033, max_age 3083 00000100:00080000:0.0:1788201623.893620:0:42607:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != REPLAY) req@ffff8e828793dc00 x1875062187582976/t0(0) o41->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201623.893631:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00000100:00080000:1.0:1788201624.069029:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201624.069037:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:1.0:1788201624.069080:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187583104:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201624.069173:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187583360:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201624.069211:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788201624.069217:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187583232:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201624.069241:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187583488:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201624.073064:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187583232:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201624.073078:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187583488:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201624.073087:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187583104:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201624.073096:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187583360:192.168.201.121@tcp:400:kworker.0 00010000:00010000:0.0:1788201624.776066:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201624.901194:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:2.0:1788201625.796715:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201625.924447:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201626.820778:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201626.948929:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201627.845374:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201627.972452:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:2.0:1788201628.868498:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201628.996575:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00000100:00100000:2.0:1788201629.190693:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2404:1875062187583616:192.168.201.121@tcp:400:kworker.0 00000100:00080000:3.0:1788201629.190862:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201629.190867:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:2.0:1788201629.192337:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788201629.192354:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187583744:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201629.192625:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187584000:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201629.192661:0:2403:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788201629.192667:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187583872:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201629.197135:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2404:1875062187583616:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201629.202143:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187583872:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201629.202144:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187583744:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201629.202159:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187584000:192.168.201.121@tcp:400:kworker.0 00010000:00010000:0.0:1788201629.892553:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201630.020466:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201630.916548:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201631.044425:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:2.0:1788201631.943736:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201632.068436:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201632.969220:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201633.092765:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201633.988670:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201634.116617:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00000100:00080000:1.0:1788201634.308922:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201634.308929:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:1.0:1788201634.308978:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187584256:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201634.309078:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187584512:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201634.309109:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788201634.309115:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187584128:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201634.309143:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187584384:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201634.315212:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187584128:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201634.315235:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187584384:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201634.315258:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187584256:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201634.315268:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187584512:192.168.201.121@tcp:400:kworker.0 00010000:00010000:2.0:1788201635.012710:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201635.140434:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201636.036572:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201636.164449:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201637.060738:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201637.188466:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:2.0:1788201638.084711:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201638.212575:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201639.109111:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201639.236469:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00000100:00080000:1.0:1788201639.428951:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201639.428960:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:0.0:1788201639.429012:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187584640:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201639.429012:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187584768:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201639.429347:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187585024:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201639.435239:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187585024:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201639.435932:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187584768:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201639.445514:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187584896:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201639.468182:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187584640:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201639.468211:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187584896:192.168.201.121@tcp:400:kworker.0 00010000:00010000:0.0:1788201640.136000:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201640.261981:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:2.0:1788201641.157032:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201641.285906:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201642.180878:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201642.309192:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201643.204748:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201643.332481:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:2.0:1788201644.228769:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201644.356473:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00000100:00080000:1.0:1788201644.548544:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201644.548552:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:1.0:1788201644.548611:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187585280:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201644.548712:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187585536:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201644.548744:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788201644.548750:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187585152:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201644.548774:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187585408:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201644.558222:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187585152:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201644.558246:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187585408:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201644.558258:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187585280:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201644.558280:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187585536:192.168.201.121@tcp:400:kworker.0 00010000:00010000:0.0:1788201645.252866:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201645.383490:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201646.276787:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201646.404580:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:2.0:1788201647.303869:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201647.428834:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201648.324881:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201648.452474:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201649.348744:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201649.483740:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00000100:00080000:1.0:1788201649.668611:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201649.668620:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:1.0:1788201649.668703:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187585792:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201649.668862:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187586048:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201649.668903:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788201649.668910:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187585664:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201649.668941:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187585920:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201649.678520:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187585664:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201649.678551:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187585920:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201649.678564:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187585792:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201649.678574:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187586048:192.168.201.121@tcp:400:kworker.0 00010000:00010000:2.0:1788201650.372538:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201650.501933:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201651.396998:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201651.524597:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201652.420616:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201652.548497:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:2.0:1788201653.444565:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201653.572927:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201654.468571:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201654.596923:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00000100:00080000:3.0:1788201654.788630:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201654.788639:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:2.0:1788201654.788686:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2404:1875062187586176:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201654.788723:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187586304:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201654.788834:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187586560:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201654.788885:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187586432:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201654.799872:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187586432:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201654.799899:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2404:1875062187586176:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201654.800621:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187586560:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201654.800833:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187586304:192.168.201.121@tcp:400:kworker.0 00010000:00010000:0.0:1788201655.502780:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201655.620520:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:2.0:1788201656.516738:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201656.644534:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201657.543016:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201657.668719:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201658.564694:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201658.693138:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:1.0:1788201659.589481:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201659.716455:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00000100:00080000:1.0:1788201659.908924:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201659.908933:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:0.0:1788201659.909281:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187586688:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201659.909532:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187586816:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201659.909786:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187587072:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201659.910125:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187586944:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201659.914228:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187587072:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201659.916915:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187586688:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201659.916945:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187586944:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201659.917019:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187586816:192.168.201.121@tcp:400:kworker.0 00010000:00010000:0.0:1788201660.612647:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201660.740477:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201661.636940:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201661.766034:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:1.0:1788201662.660750:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201662.788439:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201663.684577:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201663.812453:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201664.708593:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201664.836440:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00000100:00080000:2.0:1788201665.028718:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201665.028733:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:3.0:1788201665.028768:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2404:1875062187587200:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201665.028777:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187587328:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201665.028866:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187587456:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201665.029827:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187587584:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201665.036635:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2404:1875062187587200:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201665.036661:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187587328:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201665.036672:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187587456:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201665.044892:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187587584:192.168.201.121@tcp:400:kworker.0 00010000:00010000:1.0:1788201665.732643:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201665.860460:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201666.759159:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201666.884566:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201667.783059:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201667.908613:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:1.0:1788201668.806115:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201668.932490:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201669.829345:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201669.956489:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00000100:00080000:1.0:1788201670.150984:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201670.150991:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:0.0:1788201670.151104:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187587712:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201670.151224:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187587968:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201670.151266:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1788201670.151276:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187587840:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201670.151303:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187588096:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201670.155014:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187587840:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201670.155034:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187588096:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201670.155045:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187587712:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201670.155056:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187587968:192.168.201.121@tcp:400:kworker.0 00010000:00010000:0.0:1788201670.853114:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201670.980438:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:1.0:1788201671.884926:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201672.004474:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201672.901053:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201673.028501:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201673.924538:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201674.052696:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:1.0:1788201674.952408:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201675.076441:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00000100:00080000:0.0:1788201675.270513:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201675.270520:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:1.0:1788201675.270528:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:0.0:1788201675.270561:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187588480:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201675.270563:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187588352:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201675.270751:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187588224:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201675.272727:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:0.0:1788201675.272742:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187588608:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201675.278076:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187588480:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201675.278476:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187588352:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201675.280890:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187588224:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201675.285505:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187588608:192.168.201.121@tcp:400:kworker.0 00010000:00010000:0.0:1788201675.973033:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201676.100821:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201676.996609:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201677.124462:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:1.0:1788201678.020516:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201678.148444:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201679.045336:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201679.172437:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201680.073136:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201680.196790:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00000100:00080000:1.0:1788201680.388751:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201680.388758:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:0.0:1788201680.388869:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187588736:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201680.388971:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187588992:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201680.389006:0:2402:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:0.0:1788201680.389013:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187588864:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201680.389036:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187589120:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201680.391930:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187589120:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201680.391948:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187588992:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201680.392737:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187588864:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201680.393344:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187588736:192.168.201.121@tcp:400:kworker.0 00010000:00010000:1.0:1788201681.097266:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201681.220524:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:0.0:1788201682.116563:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201682.245757:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:1.0:1788201683.140599:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201683.268431:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00010000:00010000:1.0:1788201684.164810:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201684.292483:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 0 seconds 00000100:00080000:3.0:1788201684.532818:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201684.532839:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201684.532842:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:3.0:1788201684.532848:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201684.532857:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378726 00000100:00080000:3.0:1788201684.532868:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d717480 x1875062187538432/t81604378727(81604378727) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201589 ref 1 fl New:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201684.532881:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187537920:192.168.201.121@tcp:36:mcreate.0 00000100:00100000:3.0:1788201684.532908:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d717480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187538432:192.168.201.121@tcp:36:rm.0 00000100:00080000:3.0:1788201684.547993:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d717480 x1875062187538432/t81604378727(81604378727) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 512/456 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201684.548012:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201684.548016:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:3.0:1788201684.548023:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201684.548039:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378727 00000100:00080000:3.0:1788201684.548048:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289512680 x1875062187541504/t81604378731(81604378731) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201684.548062:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d717480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187538432:192.168.201.121@tcp:36:rm.0 00000100:00100000:3.0:1788201684.548089:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187541504:192.168.201.121@tcp:101:createmany.0 00000100:00080000:3.0:1788201684.553260:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289512680 x1875062187541504/t81604378731(81604378731) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:3.0:1788201684.553287:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d717100 x1875062187541888/t81604378732(81604378732) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:3.0:1788201684.553297:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289512680 x1875062187541504/t81604378731(81604378731) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201684.572838:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201684.572848:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:3.0:1788201684.572865:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201684.572876:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378731 00000100:00080000:3.0:1788201684.572885:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d717100 x1875062187541888/t81604378732(81604378732) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201684.572898:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187541504:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.575284:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d717100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187541888:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.586173:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d717100 x1875062187541888/t81604378732(81604378732) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.586188:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.586192:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.586198:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.586207:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378732 00000100:00080000:2.0:1788201684.586215:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d717b80 x1875062187542272/t81604378733(81604378733) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.586227:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d717100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187541888:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.586247:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d717b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187542272:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.593747:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d717b80 x1875062187542272/t81604378733(81604378733) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.593773:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d715f80 x1875062187542528/t81604378734(81604378734) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.593781:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d717b80 x1875062187542272/t81604378733(81604378733) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.593789:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.593792:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.593798:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.593805:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378733 00000100:00080000:2.0:1788201684.593814:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d715f80 x1875062187542528/t81604378734(81604378734) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.593826:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d717b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187542272:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.593848:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d715f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187542528:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.599069:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d715f80 x1875062187542528/t81604378734(81604378734) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.599085:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.599088:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.599095:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.599103:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378734 00000100:00080000:2.0:1788201684.599112:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d714700 x1875062187542912/t81604378735(81604378735) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.599132:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d715f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187542528:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.599150:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d714700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187542912:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.603093:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d714700 x1875062187542912/t81604378735(81604378735) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.603116:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d6a4000 x1875062187543168/t81604378736(81604378736) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.603125:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d714700 x1875062187542912/t81604378735(81604378735) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.603134:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.603136:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.603141:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.603149:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378735 00000100:00080000:2.0:1788201684.603157:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d6a4000 x1875062187543168/t81604378736(81604378736) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.603168:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d714700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187542912:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.603184:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d6a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187543168:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.605273:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d6a4000 x1875062187543168/t81604378736(81604378736) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.605285:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.605288:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.605294:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.605301:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378736 00000100:00080000:2.0:1788201684.605340:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d6a6a00 x1875062187543552/t81604378737(81604378737) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.605356:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187543168:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.605373:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d6a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187543552:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.611027:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d6a6a00 x1875062187543552/t81604378737(81604378737) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.611048:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d6a4380 x1875062187543808/t81604378738(81604378738) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.611055:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d6a6a00 x1875062187543552/t81604378737(81604378737) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.611063:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.611066:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.611072:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.611080:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378737 00000100:00080000:2.0:1788201684.611088:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d6a4380 x1875062187543808/t81604378738(81604378738) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.611101:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187543552:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.611120:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d6a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187543808:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.613118:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d6a4380 x1875062187543808/t81604378738(81604378738) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.613132:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.613134:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.613140:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.613147:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378738 00000100:00080000:2.0:1788201684.613155:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d716d80 x1875062187544192/t81604378739(81604378739) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.613166:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187543808:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.613182:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d716d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187544192:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.620128:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d716d80 x1875062187544192/t81604378739(81604378739) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.620152:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d715c00 x1875062187544448/t81604378740(81604378740) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.620161:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d716d80 x1875062187544192/t81604378739(81604378739) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.620169:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.620172:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.620177:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.620184:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378739 00000100:00080000:2.0:1788201684.620193:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d715c00 x1875062187544448/t81604378740(81604378740) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.620205:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187544192:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.620224:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d715c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187544448:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.623096:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d715c00 x1875062187544448/t81604378740(81604378740) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.623111:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.623113:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.623119:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.623126:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378740 00000100:00080000:2.0:1788201684.623135:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d714380 x1875062187544832/t81604378741(81604378741) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.623146:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d715c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187544448:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.623165:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d714380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187544832:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.626183:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d714380 x1875062187544832/t81604378741(81604378741) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.626204:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d716a00 x1875062187545088/t81604378742(81604378742) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.626212:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d714380 x1875062187544832/t81604378741(81604378741) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.626220:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.626223:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.626227:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.626233:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378741 00000100:00080000:2.0:1788201684.626241:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d716a00 x1875062187545088/t81604378742(81604378742) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.626251:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d714380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187544832:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.626269:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d716a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187545088:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.628211:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d716a00 x1875062187545088/t81604378742(81604378742) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.628224:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.628226:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.628230:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.628237:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378742 00000100:00080000:2.0:1788201684.628245:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289510000 x1875062187545472/t81604378743(81604378743) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.628254:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187545088:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.628269:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187545472:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.631138:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289510000 x1875062187545472/t81604378743(81604378743) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.631160:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e8289511f80 x1875062187545728/t81604378744(81604378744) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.631170:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289510000 x1875062187545472/t81604378743(81604378743) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.631179:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.631181:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.631187:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.631196:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378743 00000100:00080000:2.0:1788201684.631204:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289511f80 x1875062187545728/t81604378744(81604378744) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.631215:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187545472:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.631232:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187545728:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.634216:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289511f80 x1875062187545728/t81604378744(81604378744) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.634230:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.634233:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.634238:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.634247:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378744 00000100:00080000:2.0:1788201684.634256:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289513b80 x1875062187546112/t81604378745(81604378745) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.634269:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187545728:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.634286:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187546112:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.637926:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289513b80 x1875062187546112/t81604378745(81604378745) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.637948:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e8289510a80 x1875062187546368/t81604378746(81604378746) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.637957:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289513b80 x1875062187546112/t81604378745(81604378745) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.637965:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.637967:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.637973:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.637980:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378745 00000100:00080000:2.0:1788201684.637990:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289510a80 x1875062187546368/t81604378746(81604378746) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.638002:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289513b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187546112:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.638021:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187546368:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.640290:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289510a80 x1875062187546368/t81604378746(81604378746) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.640302:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.640340:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.640348:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.640355:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378746 00000100:00080000:2.0:1788201684.640364:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289510700 x1875062187546752/t81604378747(81604378747) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.640374:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187546368:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.640390:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289510700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187546752:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.643942:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289510700 x1875062187546752/t81604378747(81604378747) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.643964:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e8289512d80 x1875062187547008/t81604378748(81604378748) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.643974:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289510700 x1875062187546752/t81604378747(81604378747) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.643982:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.643986:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.643992:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.644000:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378747 00000100:00080000:2.0:1788201684.644010:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289512d80 x1875062187547008/t81604378748(81604378748) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.644021:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289510700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187546752:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.644039:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187547008:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.649146:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289512d80 x1875062187547008/t81604378748(81604378748) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.649161:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.649165:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.649171:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.649178:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378748 00000100:00080000:2.0:1788201684.649190:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289512300 x1875062187547392/t81604378749(81604378749) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.649200:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187547008:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.649221:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289512300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187547392:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.652372:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289512300 x1875062187547392/t81604378749(81604378749) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.652396:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e8289513100 x1875062187547648/t81604378750(81604378750) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.652406:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289512300 x1875062187547392/t81604378749(81604378749) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.652414:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.652417:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.652423:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.652430:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378749 00000100:00080000:2.0:1788201684.652439:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289513100 x1875062187547648/t81604378750(81604378750) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.652451:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289512300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187547392:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.652471:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289513100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187547648:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.655779:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289513100 x1875062187547648/t81604378750(81604378750) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.655793:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.655796:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.655803:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.655812:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378750 00000100:00080000:2.0:1788201684.655822:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d6a6d80 x1875062187548032/t81604378751(81604378751) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.655834:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289513100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187547648:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.655852:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d6a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187548032:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.659454:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d6a6d80 x1875062187548032/t81604378751(81604378751) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.659479:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d6a4700 x1875062187548288/t81604378752(81604378752) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.659548:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d6a6d80 x1875062187548032/t81604378751(81604378751) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.659615:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.659618:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.659691:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.659700:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378751 00000100:00080000:2.0:1788201684.659712:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d6a4700 x1875062187548288/t81604378752(81604378752) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.659724:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187548032:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.659743:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d6a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187548288:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.662248:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d6a4700 x1875062187548288/t81604378752(81604378752) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.662262:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.662265:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.662270:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.662277:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378752 00000100:00080000:2.0:1788201684.662287:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d6a5f80 x1875062187548672/t81604378753(81604378753) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.662298:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187548288:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.662338:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d6a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187548672:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.665511:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d6a5f80 x1875062187548672/t81604378753(81604378753) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.665531:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d6a7b80 x1875062187548928/t81604378754(81604378754) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.665541:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d6a5f80 x1875062187548672/t81604378753(81604378753) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.665548:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.665602:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.665703:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.665711:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378753 00000100:00080000:2.0:1788201684.665721:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d6a7b80 x1875062187548928/t81604378754(81604378754) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.665732:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187548672:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.665749:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d6a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187548928:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.668226:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d6a7b80 x1875062187548928/t81604378754(81604378754) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.668237:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.668240:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.668245:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.668252:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378754 00000100:00080000:2.0:1788201684.668261:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d6a4a80 x1875062187549312/t81604378755(81604378755) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.668271:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187548928:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.668287:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d6a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187549312:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.673162:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d6a4a80 x1875062187549312/t81604378755(81604378755) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.673183:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d6a7100 x1875062187549568/t81604378756(81604378756) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.673193:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d6a4a80 x1875062187549312/t81604378755(81604378755) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.673201:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.673204:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.673210:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.673218:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378755 00000100:00080000:2.0:1788201684.673229:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d6a7100 x1875062187549568/t81604378756(81604378756) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.673242:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187549312:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.673262:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d6a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187549568:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.676302:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d6a7100 x1875062187549568/t81604378756(81604378756) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.676346:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.676349:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.676355:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.676362:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378756 00000100:00080000:2.0:1788201684.676373:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d6a5500 x1875062187549952/t81604378757(81604378757) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.676385:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187549568:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.676406:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d6a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187549952:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.682559:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d6a5500 x1875062187549952/t81604378757(81604378757) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.682581:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d6a7480 x1875062187550208/t81604378758(81604378758) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.682590:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d6a5500 x1875062187549952/t81604378757(81604378757) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.682598:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.682601:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.682607:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.682614:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378757 00000100:00080000:2.0:1788201684.682738:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d6a7480 x1875062187550208/t81604378758(81604378758) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.682753:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187549952:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.682772:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d6a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187550208:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.686142:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d6a7480 x1875062187550208/t81604378758(81604378758) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.686156:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.686159:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.686164:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.686170:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378758 00000100:00080000:2.0:1788201684.686182:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d6a6300 x1875062187550592/t81604378759(81604378759) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.686192:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187550208:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.686210:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d6a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187550592:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.693092:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d6a6300 x1875062187550592/t81604378759(81604378759) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.693114:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d6a7800 x1875062187550848/t81604378760(81604378760) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.693122:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d6a6300 x1875062187550592/t81604378759(81604378759) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.693129:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.693132:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.693137:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.693145:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378759 00000100:00080000:2.0:1788201684.693155:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d6a7800 x1875062187550848/t81604378760(81604378760) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.693165:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187550592:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.693184:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d6a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187550848:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.698398:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d6a7800 x1875062187550848/t81604378760(81604378760) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.698411:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.698415:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.698422:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.698430:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378760 00000100:00080000:2.0:1788201684.698442:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d6a5c00 x1875062187551232/t81604378761(81604378761) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.698453:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187550848:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.698472:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d6a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187551232:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.703464:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d6a5c00 x1875062187551232/t81604378761(81604378761) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.703493:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d6a6680 x1875062187551488/t81604378762(81604378762) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.703502:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d6a5c00 x1875062187551232/t81604378761(81604378761) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.703510:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.703513:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.703518:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.703526:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378761 00000100:00080000:2.0:1788201684.703538:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d6a6680 x1875062187551488/t81604378762(81604378762) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.703549:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187551232:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.703569:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d6a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187551488:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.707287:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d6a6680 x1875062187551488/t81604378762(81604378762) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.707301:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.707305:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.707414:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.707423:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378762 00000100:00080000:2.0:1788201684.707434:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d6a5180 x1875062187551872/t81604378763(81604378763) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.707447:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187551488:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.707468:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d6a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187551872:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.711885:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d6a5180 x1875062187551872/t81604378763(81604378763) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.711906:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d715880 x1875062187552128/t81604378764(81604378764) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.711914:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d6a5180 x1875062187551872/t81604378763(81604378763) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.711922:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.711925:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.711931:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.711938:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378763 00000100:00080000:2.0:1788201684.711949:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d715880 x1875062187552128/t81604378764(81604378764) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.711961:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187551872:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.711979:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d715880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187552128:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.716397:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d715880 x1875062187552128/t81604378764(81604378764) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.716411:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.716414:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.716420:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.716428:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378764 00000100:00080000:2.0:1788201684.716441:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828793df80 x1875062187552512/t81604378765(81604378765) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.716453:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d715880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187552128:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.716474:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187552512:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.720103:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828793df80 x1875062187552512/t81604378765(81604378765) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.720126:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828793d880 x1875062187552768/t81604378766(81604378766) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.720136:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828793df80 x1875062187552512/t81604378765(81604378765) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.720143:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.720147:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.720152:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.720159:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378765 00000100:00080000:2.0:1788201684.720171:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828793d880 x1875062187552768/t81604378766(81604378766) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.720182:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187552512:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.720201:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187552768:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.723301:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828793d880 x1875062187552768/t81604378766(81604378766) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.723338:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.723341:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.723348:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.723356:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378766 00000100:00080000:2.0:1788201684.723368:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828793ce00 x1875062187553152/t81604378767(81604378767) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.723380:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187552768:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.723399:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187553152:192.168.201.121@tcp:101:createmany.0 00000100:00080000:3.0:1788201684.728218:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828793ce00 x1875062187553152/t81604378767(81604378767) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:3.0:1788201684.728243:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d6a4e00 x1875062187553408/t81604378768(81604378768) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:3.0:1788201684.728250:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828793ce00 x1875062187553152/t81604378767(81604378767) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201684.728257:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201684.728259:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:3.0:1788201684.728264:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201684.728269:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378767 00000100:00080000:3.0:1788201684.728279:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d6a4e00 x1875062187553408/t81604378768(81604378768) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201684.728288:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187553152:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201684.728331:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d6a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187553408:192.168.201.121@tcp:35:createmany.0 00000100:00080000:3.0:1788201684.731638:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d6a4e00 x1875062187553408/t81604378768(81604378768) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201684.731681:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201684.731684:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:3.0:1788201684.731692:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201684.731700:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378768 00000100:00080000:3.0:1788201684.731712:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289551180 x1875062187553792/t81604378769(81604378769) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201684.731729:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d6a4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187553408:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201684.731748:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289551180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187553792:192.168.201.121@tcp:101:createmany.0 00000100:00080000:3.0:1788201684.740436:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289551180 x1875062187553792/t81604378769(81604378769) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:3.0:1788201684.740458:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e8289553b80 x1875062187554048/t81604378770(81604378770) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:3.0:1788201684.740467:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289551180 x1875062187553792/t81604378769(81604378769) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201684.740475:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201684.740477:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:3.0:1788201684.740495:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201684.740504:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378769 00000100:00080000:3.0:1788201684.740517:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289553b80 x1875062187554048/t81604378770(81604378770) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201684.740529:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289551180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187553792:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201684.740548:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289553b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187554048:192.168.201.121@tcp:35:createmany.0 00000100:00080000:3.0:1788201684.744062:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289553b80 x1875062187554048/t81604378770(81604378770) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201684.744076:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201684.744078:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:3.0:1788201684.744084:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201684.744091:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378770 00000100:00080000:3.0:1788201684.744103:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289550e00 x1875062187554432/t81604378771(81604378771) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201684.744114:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289553b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187554048:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201684.744132:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187554432:192.168.201.121@tcp:101:createmany.0 00000100:00080000:3.0:1788201684.748384:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289550e00 x1875062187554432/t81604378771(81604378771) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:3.0:1788201684.748406:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828793c380 x1875062187554688/t81604378772(81604378772) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:3.0:1788201684.748415:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289550e00 x1875062187554432/t81604378771(81604378771) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201684.748422:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201684.748425:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:3.0:1788201684.748430:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201684.748438:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378771 00000100:00080000:3.0:1788201684.748449:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828793c380 x1875062187554688/t81604378772(81604378772) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201684.748461:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289550e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187554432:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201684.748490:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187554688:192.168.201.121@tcp:35:createmany.0 00000100:00080000:3.0:1788201684.751536:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828793c380 x1875062187554688/t81604378772(81604378772) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201684.751549:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201684.751724:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:3.0:1788201684.751732:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201684.751741:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378772 00000100:00080000:3.0:1788201684.751754:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d716680 x1875062187555072/t81604378773(81604378773) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201684.751766:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187554688:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201684.751784:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187555072:192.168.201.121@tcp:101:createmany.0 00000100:00080000:3.0:1788201684.756184:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d716680 x1875062187555072/t81604378773(81604378773) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:3.0:1788201684.756209:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d714e00 x1875062187555328/t81604378774(81604378774) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:3.0:1788201684.756219:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d716680 x1875062187555072/t81604378773(81604378773) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201684.756227:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201684.756230:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:3.0:1788201684.756235:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201684.756243:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378773 00000100:00080000:3.0:1788201684.756256:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d714e00 x1875062187555328/t81604378774(81604378774) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201684.756267:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187555072:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201684.756286:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d714e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187555328:192.168.201.121@tcp:35:createmany.0 00000100:00080000:3.0:1788201684.760350:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d714e00 x1875062187555328/t81604378774(81604378774) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201684.760365:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201684.760368:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:3.0:1788201684.760374:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201684.760381:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378774 00000100:00080000:3.0:1788201684.760395:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d714a80 x1875062187555712/t81604378775(81604378775) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201684.760406:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d714e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187555328:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201684.760424:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d714a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187555712:192.168.201.121@tcp:101:createmany.0 00000100:00080000:3.0:1788201684.764439:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d714a80 x1875062187555712/t81604378775(81604378775) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:3.0:1788201684.764462:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d714000 x1875062187555968/t81604378776(81604378776) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:3.0:1788201684.764471:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d714a80 x1875062187555712/t81604378775(81604378775) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201684.764492:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201684.764497:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:3.0:1788201684.764505:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201684.764512:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378775 00000100:00080000:3.0:1788201684.764526:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d714000 x1875062187555968/t81604378776(81604378776) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201684.764537:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d714a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187555712:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201684.764742:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d714000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187555968:192.168.201.121@tcp:35:createmany.0 00000100:00080000:3.0:1788201684.767866:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d714000 x1875062187555968/t81604378776(81604378776) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201684.767880:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201684.767883:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:3.0:1788201684.767888:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201684.767895:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378776 00000100:00080000:3.0:1788201684.767908:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d716300 x1875062187556352/t81604378777(81604378777) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201684.767920:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d714000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187555968:192.168.201.121@tcp:35:createmany.0 00000100:00100000:3.0:1788201684.767938:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d716300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187556352:192.168.201.121@tcp:101:createmany.0 00000100:00080000:3.0:1788201684.772104:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d716300 x1875062187556352/t81604378777(81604378777) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:3.0:1788201684.772128:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d717800 x1875062187556608/t81604378778(81604378778) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:3.0:1788201684.772137:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d716300 x1875062187556352/t81604378777(81604378777) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201684.772145:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201684.772148:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:3.0:1788201684.772153:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201684.772160:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378777 00000100:00080000:3.0:1788201684.772173:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d717800 x1875062187556608/t81604378778(81604378778) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201684.772184:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d716300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187556352:192.168.201.121@tcp:101:createmany.0 00000100:00100000:3.0:1788201684.772202:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d717800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187556608:192.168.201.121@tcp:35:createmany.0 00000100:00080000:3.0:1788201684.775299:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d717800 x1875062187556608/t81604378778(81604378778) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:3.0:1788201684.775396:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788201684.775399:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:3.0:1788201684.775405:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201684.775412:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378778 00000100:00080000:3.0:1788201684.775425:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d715180 x1875062187556992/t81604378779(81604378779) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788201684.775436:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d717800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187556608:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.775700:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d715180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187556992:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.779475:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d715180 x1875062187556992/t81604378779(81604378779) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.779775:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e8289552680 x1875062187557248/t81604378780(81604378780) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.779783:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d715180 x1875062187556992/t81604378779(81604378779) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.779791:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.779794:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.779799:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.779807:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378779 00000100:00080000:2.0:1788201684.779821:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289552680 x1875062187557248/t81604378780(81604378780) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.779832:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d715180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187556992:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.779851:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289552680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187557248:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.782890:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289552680 x1875062187557248/t81604378780(81604378780) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.782904:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.782907:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.782912:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.782932:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378780 00000100:00080000:2.0:1788201684.782945:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828953b480 x1875062187557888/t81604378781(81604378781) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.782962:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289552680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187557248:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.782980:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187557888:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.787251:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828953b480 x1875062187557888/t81604378781(81604378781) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.787272:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828953a300 x1875062187558144/t81604378782(81604378782) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.787281:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828953b480 x1875062187557888/t81604378781(81604378781) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.787289:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.787291:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.787297:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.787303:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378781 00000100:00080000:2.0:1788201684.787342:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828953a300 x1875062187558144/t81604378782(81604378782) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.787354:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187557888:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.787373:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187558144:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.790585:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828953a300 x1875062187558144/t81604378782(81604378782) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.790599:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.790602:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.790609:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.790617:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378782 00000100:00080000:2.0:1788201684.790749:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289551880 x1875062187558528/t81604378783(81604378783) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.790762:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187558144:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.790779:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187558528:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.794226:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289551880 x1875062187558528/t81604378783(81604378783) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.794246:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e8289550000 x1875062187558784/t81604378784(81604378784) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.794255:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289551880 x1875062187558528/t81604378783(81604378783) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.794262:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.794265:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.794270:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.794277:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378783 00000100:00080000:2.0:1788201684.794291:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289550000 x1875062187558784/t81604378784(81604378784) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.794302:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289551880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187558528:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.794348:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289550000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187558784:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.796920:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289550000 x1875062187558784/t81604378784(81604378784) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.796932:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.796935:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.796940:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.796947:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378784 00000100:00080000:2.0:1788201684.796961:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289552a00 x1875062187559168/t81604378785(81604378785) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.796973:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289550000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187558784:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.796990:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289552a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187559168:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.801515:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289552a00 x1875062187559168/t81604378785(81604378785) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.801536:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e8289550380 x1875062187559424/t81604378786(81604378786) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.801545:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289552a00 x1875062187559168/t81604378785(81604378785) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.801559:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.801562:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.801568:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.801575:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378785 00000100:00080000:2.0:1788201684.801588:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289550380 x1875062187559424/t81604378786(81604378786) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.801600:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289552a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187559168:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.801618:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289550380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187559424:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.804819:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289550380 x1875062187559424/t81604378786(81604378786) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.804833:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.804837:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.804842:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.804851:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378786 00000100:00080000:2.0:1788201684.804867:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289551c00 x1875062187559808/t81604378787(81604378787) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.804879:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289550380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187559424:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.804898:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289551c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187559808:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.812356:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289551c00 x1875062187559808/t81604378787(81604378787) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.812379:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e8289552d80 x1875062187560064/t81604378788(81604378788) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.812388:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289551c00 x1875062187559808/t81604378787(81604378787) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.812396:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.812399:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.812405:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.812412:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378787 00000100:00080000:2.0:1788201684.812429:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289552d80 x1875062187560064/t81604378788(81604378788) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.812441:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289551c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187559808:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.812460:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289552d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187560064:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.814891:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289552d80 x1875062187560064/t81604378788(81604378788) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.814904:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.814907:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.814913:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.814920:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378788 00000100:00080000:2.0:1788201684.814933:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289550700 x1875062187560448/t81604378789(81604378789) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.814944:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289552d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187560064:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.814960:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289550700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187560448:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.818769:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289550700 x1875062187560448/t81604378789(81604378789) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.818790:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e8289551f80 x1875062187560704/t81604378790(81604378790) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.818800:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289550700 x1875062187560448/t81604378789(81604378789) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.818808:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.818811:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.818817:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.818824:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378789 00000100:00080000:2.0:1788201684.818838:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289551f80 x1875062187560704/t81604378790(81604378790) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.818849:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289550700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187560448:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.818870:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289551f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187560704:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.822405:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289551f80 x1875062187560704/t81604378790(81604378790) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.822421:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.822425:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.822431:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.822439:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378790 00000100:00080000:2.0:1788201684.822455:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289550a80 x1875062187561088/t81604378791(81604378791) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.822467:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289551f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187560704:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.822496:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289550a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187561088:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.826536:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289550a80 x1875062187561088/t81604378791(81604378791) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.826626:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e8289553100 x1875062187561344/t81604378792(81604378792) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.826637:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289550a80 x1875062187561088/t81604378791(81604378791) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.826645:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.826656:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.826662:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.826670:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378791 00000100:00080000:2.0:1788201684.826689:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289553100 x1875062187561344/t81604378792(81604378792) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.826700:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289550a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187561088:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.826720:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187561344:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.830301:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289553100 x1875062187561344/t81604378792(81604378792) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.830340:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.830343:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.830349:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.830357:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378792 00000100:00080000:2.0:1788201684.830375:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289551500 x1875062187561728/t81604378793(81604378793) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.830387:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289553100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187561344:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.830408:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187561728:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.834271:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289551500 x1875062187561728/t81604378793(81604378793) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.834293:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e8289553480 x1875062187561984/t81604378794(81604378794) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.834303:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289551500 x1875062187561728/t81604378793(81604378793) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.834331:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.834333:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.834339:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.834347:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378793 00000100:00080000:2.0:1788201684.834362:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289553480 x1875062187561984/t81604378794(81604378794) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.834374:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289551500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187561728:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.834391:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187561984:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.837410:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289553480 x1875062187561984/t81604378794(81604378794) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.837424:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.837427:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.837433:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.837440:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378794 00000100:00080000:2.0:1788201684.837454:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289552300 x1875062187562368/t81604378795(81604378795) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.837466:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289553480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187561984:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.837540:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289552300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187562368:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.840715:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289552300 x1875062187562368/t81604378795(81604378795) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.840736:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e8289553800 x1875062187562624/t81604378796(81604378796) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.840745:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289552300 x1875062187562368/t81604378795(81604378795) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.840753:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.840755:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.840760:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.840768:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378795 00000100:00080000:2.0:1788201684.840784:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289553800 x1875062187562624/t81604378796(81604378796) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.840796:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289552300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187562368:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.840817:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289553800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187562624:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.843328:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289553800 x1875062187562624/t81604378796(81604378796) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.843343:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.843346:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.843353:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.843360:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378796 00000100:00080000:2.0:1788201684.843377:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d701880 x1875062187563008/t81604378797(81604378797) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.843387:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289553800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187562624:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.843405:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d701880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187563008:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.847727:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d701880 x1875062187563008/t81604378797(81604378797) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.847765:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d701180 x1875062187563264/t81604378798(81604378798) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.847775:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d701880 x1875062187563008/t81604378797(81604378797) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.847783:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.847786:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.847791:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.847798:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378797 00000100:00080000:2.0:1788201684.847824:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d701180 x1875062187563264/t81604378798(81604378798) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.847834:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d701880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187563008:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.847862:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d701180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187563264:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.851167:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d701180 x1875062187563264/t81604378798(81604378798) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.851180:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.851183:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.851189:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.851196:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378798 00000100:00080000:2.0:1788201684.851212:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d703800 x1875062187563648/t81604378799(81604378799) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.851222:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d701180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187563264:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.851239:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d703800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187563648:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.854031:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d703800 x1875062187563648/t81604378799(81604378799) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.854053:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d700e00 x1875062187563904/t81604378800(81604378800) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.854062:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d703800 x1875062187563648/t81604378799(81604378799) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.854070:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.854073:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.854078:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.854086:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378799 00000100:00080000:2.0:1788201684.854100:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d700e00 x1875062187563904/t81604378800(81604378800) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.854109:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d703800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187563648:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.854126:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d700e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187563904:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.856424:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d700e00 x1875062187563904/t81604378800(81604378800) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.856438:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.856442:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.856447:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.856454:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378800 00000100:00080000:2.0:1788201684.856470:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d702680 x1875062187564288/t81604378801(81604378801) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.856481:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d700e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187563904:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.856502:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d702680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187564288:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.859899:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d702680 x1875062187564288/t81604378801(81604378801) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.859922:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d700000 x1875062187564544/t81604378802(81604378802) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.859932:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d702680 x1875062187564288/t81604378801(81604378801) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.859939:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.859942:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.859947:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.859955:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378801 00000100:00080000:2.0:1788201684.859973:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d700000 x1875062187564544/t81604378802(81604378802) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.859984:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d702680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187564288:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.860070:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d700000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187564544:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.862399:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d700000 x1875062187564544/t81604378802(81604378802) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.862410:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.862412:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.862416:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.862421:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378802 00000100:00080000:2.0:1788201684.862434:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d702a00 x1875062187564928/t81604378803(81604378803) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.862441:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d700000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187564544:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.862455:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d702a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187564928:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.865360:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d702a00 x1875062187564928/t81604378803(81604378803) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.865381:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d700380 x1875062187565184/t81604378804(81604378804) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.865390:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d702a00 x1875062187564928/t81604378803(81604378803) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.865396:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.865399:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.865403:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.865408:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378803 00000100:00080000:2.0:1788201684.865420:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d700380 x1875062187565184/t81604378804(81604378804) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.865428:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d702a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187564928:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.865442:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d700380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187565184:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.868355:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d700380 x1875062187565184/t81604378804(81604378804) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.868366:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.868367:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.868371:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.868376:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378804 00000100:00080000:2.0:1788201684.868391:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828953b800 x1875062187565568/t81604378805(81604378805) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.868398:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d700380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187565184:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.868413:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187565568:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.872202:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828953b800 x1875062187565568/t81604378805(81604378805) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.872225:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e8289538e00 x1875062187565824/t81604378806(81604378806) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.872233:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828953b800 x1875062187565568/t81604378805(81604378805) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.872239:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.872242:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.872248:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.872256:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378805 00000100:00080000:2.0:1788201684.872275:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289538e00 x1875062187565824/t81604378806(81604378806) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.872452:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187565568:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.872473:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187565824:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.874759:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289538e00 x1875062187565824/t81604378806(81604378806) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.874771:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.874775:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.874780:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.874786:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378806 00000100:00080000:2.0:1788201684.874801:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289539180 x1875062187566208/t81604378807(81604378807) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.874811:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187565824:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.874826:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187566208:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.880295:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289539180 x1875062187566208/t81604378807(81604378807) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.880332:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828953a680 x1875062187566464/t81604378808(81604378808) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.880338:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289539180 x1875062187566208/t81604378807(81604378807) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.880343:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.880345:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.880349:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.880353:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378807 00000100:00080000:2.0:1788201684.880367:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828953a680 x1875062187566464/t81604378808(81604378808) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.880374:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187566208:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.880389:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187566464:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.884342:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828953a680 x1875062187566464/t81604378808(81604378808) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.884356:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.884359:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.884366:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.884374:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378808 00000100:00080000:2.0:1788201684.884391:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289539880 x1875062187566848/t81604378809(81604378809) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.884402:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187566464:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.884421:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187566848:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.894869:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289539880 x1875062187566848/t81604378809(81604378809) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.894902:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d701c00 x1875062187567104/t81604378810(81604378810) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.894911:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289539880 x1875062187566848/t81604378809(81604378809) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.894919:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.894922:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.894927:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.894934:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378809 00000100:00080000:2.0:1788201684.894952:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d701c00 x1875062187567104/t81604378810(81604378810) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.894964:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187566848:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.894983:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d701c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187567104:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.897344:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d701c00 x1875062187567104/t81604378810(81604378810) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.897359:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.897361:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.897367:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.897373:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378810 00000100:00080000:2.0:1788201684.897387:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289538000 x1875062187567488/t81604378811(81604378811) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.897397:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d701c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187567104:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.897414:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187567488:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.903017:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289538000 x1875062187567488/t81604378811(81604378811) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.903039:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d700700 x1875062187567744/t81604378812(81604378812) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.903048:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289538000 x1875062187567488/t81604378811(81604378811) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.903057:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.903061:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.903067:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.903075:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378811 00000100:00080000:2.0:1788201684.903091:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d700700 x1875062187567744/t81604378812(81604378812) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.903104:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187567488:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.903123:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d700700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187567744:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.907167:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d700700 x1875062187567744/t81604378812(81604378812) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.907182:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.907184:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.907189:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.907196:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378812 00000100:00080000:2.0:1788201684.907215:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d703b80 x1875062187568128/t81604378813(81604378813) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.907224:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d700700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187567744:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.907243:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d703b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187568128:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.914071:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d703b80 x1875062187568128/t81604378813(81604378813) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.914094:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d700a80 x1875062187568384/t81604378814(81604378814) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.914119:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d703b80 x1875062187568128/t81604378813(81604378813) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.914128:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.914131:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.914137:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.914145:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378813 00000100:00080000:2.0:1788201684.914164:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d700a80 x1875062187568384/t81604378814(81604378814) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.914175:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d703b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187568128:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.914195:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d700a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187568384:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.924384:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d700a80 x1875062187568384/t81604378814(81604378814) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.924400:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.924403:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.924410:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.924420:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378814 00000100:00080000:2.0:1788201684.924438:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d703100 x1875062187568768/t81604378815(81604378815) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.924451:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d700a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187568384:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.924471:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d703100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187568768:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.929252:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d703100 x1875062187568768/t81604378815(81604378815) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.929274:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828953aa00 x1875062187569024/t81604378816(81604378816) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.929282:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d703100 x1875062187568768/t81604378815(81604378815) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.929289:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.929292:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.929298:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.929305:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378815 00000100:00080000:2.0:1788201684.929358:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828953aa00 x1875062187569024/t81604378816(81604378816) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.929369:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d703100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187568768:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.929389:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187569024:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.938246:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828953aa00 x1875062187569024/t81604378816(81604378816) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.938260:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.938263:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.938269:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.938276:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378816 00000100:00080000:2.0:1788201684.938296:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289538380 x1875062187569408/t81604378817(81604378817) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201596 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.938350:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187569024:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.938372:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187569408:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.949538:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289538380 x1875062187569408/t81604378817(81604378817) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.949790:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e8289539c00 x1875062187569664/t81604378818(81604378818) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.949799:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289538380 x1875062187569408/t81604378817(81604378817) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.949806:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.949809:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.949814:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.949821:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378817 00000100:00080000:2.0:1788201684.949838:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289539c00 x1875062187569664/t81604378818(81604378818) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201596 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.949849:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187569408:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.949868:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187569664:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.954217:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289539c00 x1875062187569664/t81604378818(81604378818) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.954231:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.954235:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.954241:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.954249:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378818 00000100:00080000:2.0:1788201684.954267:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828793f480 x1875062187570048/t81604378819(81604378819) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201597 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.954279:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187569664:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.954297:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187570048:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.959216:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828793f480 x1875062187570048/t81604378819(81604378819) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.959238:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d701500 x1875062187570304/t81604378820(81604378820) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201597 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.959247:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828793f480 x1875062187570048/t81604378819(81604378819) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.959254:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.959257:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.959263:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.959271:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378819 00000100:00080000:2.0:1788201684.959303:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d701500 x1875062187570304/t81604378820(81604378820) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201597 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.959450:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187570048:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.959482:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d701500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187570304:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.966167:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d701500 x1875062187570304/t81604378820(81604378820) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.966182:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.966186:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.966192:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.966201:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378820 00000100:00080000:2.0:1788201684.966226:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828793fb80 x1875062187570688/t81604378821(81604378821) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201597 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.966239:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d701500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187570304:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.966261:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187570688:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.971339:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828793fb80 x1875062187570688/t81604378821(81604378821) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.971362:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d703480 x1875062187570944/t81604378822(81604378822) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201597 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.971370:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828793fb80 x1875062187570688/t81604378821(81604378821) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.971377:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.971379:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.971384:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.971391:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378821 00000100:00080000:2.0:1788201684.971410:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d703480 x1875062187570944/t81604378822(81604378822) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201597 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.971421:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187570688:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.971439:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d703480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187570944:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.974632:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d703480 x1875062187570944/t81604378822(81604378822) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.974655:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.974658:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.974663:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.974670:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378822 00000100:00080000:2.0:1788201684.974688:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d702300 x1875062187571328/t81604378823(81604378823) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201597 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.974701:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d703480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187570944:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.974719:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d702300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187571328:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.978414:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d702300 x1875062187571328/t81604378823(81604378823) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.978436:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828953ad80 x1875062187571584/t81604378824(81604378824) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201597 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.978443:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d702300 x1875062187571328/t81604378823(81604378823) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.978450:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.978454:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.978459:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.978467:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378823 00000100:00080000:2.0:1788201684.978496:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828953ad80 x1875062187571584/t81604378824(81604378824) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201597 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.978507:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d702300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187571328:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.978524:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187571584:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.982188:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828953ad80 x1875062187571584/t81604378824(81604378824) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.982202:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.982206:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.982212:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.982220:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378824 00000100:00080000:2.0:1788201684.982241:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289538700 x1875062187571968/t81604378825(81604378825) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201597 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.982255:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187571584:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.982274:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187571968:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.987046:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289538700 x1875062187571968/t81604378825(81604378825) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.987107:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828793f100 x1875062187572224/t81604378826(81604378826) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201597 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.987117:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289538700 x1875062187571968/t81604378825(81604378825) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.987124:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.987127:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.987132:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.987139:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378825 00000100:00080000:2.0:1788201684.987159:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828793f100 x1875062187572224/t81604378826(81604378826) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201597 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.987170:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187571968:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.987187:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187572224:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201684.990938:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828793f100 x1875062187572224/t81604378826(81604378826) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.990951:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.990954:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.990960:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.990967:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378826 00000100:00080000:2.0:1788201684.990987:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e8289511500 x1875062187572608/t81604378827(81604378827) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201597 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.991267:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187572224:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201684.991287:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289511500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187572608:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201684.996484:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e8289511500 x1875062187572608/t81604378827(81604378827) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201684.996507:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e828d701f80 x1875062187573888/t81604378828(81604378828) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201597 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201684.996516:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e8289511500 x1875062187572608/t81604378827(81604378827) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201700 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201684.996524:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201684.996527:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201684.996533:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201684.996540:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378827 00000100:00080000:2.0:1788201684.996561:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d701f80 x1875062187573888/t81604378828(81604378828) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201597 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201684.996571:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289511500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187572608:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201684.996590:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d701f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187573888:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201685.000420:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d701f80 x1875062187573888/t81604378828(81604378828) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201700 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201685.000434:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201685.000437:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201685.000443:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201685.000450:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378828 00000100:00080000:2.0:1788201685.000472:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e828d702d80 x1875062187574272/t81604378829(81604378829) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/664 e 0 to 0 dl 1788201597 ref 1 fl New:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201685.000614:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d701f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187573888:192.168.201.121@tcp:35:createmany.0 00000100:00100000:2.0:1788201685.000716:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d702d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187574272:192.168.201.121@tcp:101:createmany.0 00000100:00080000:2.0:1788201685.004269:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e828d702d80 x1875062187574272/t81604378829(81604378829) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201701 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788201685.004286:0:2400:0:(mdc_request.c:715:mdc_replay_open()) @@@ updating close body with new fh req@ffff8e82894d7480 x1875062187574528/t81604378830(81604378830) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201597 ref 2 fl Complete:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788201685.004291:0:2400:0:(client.c:3448:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8e828d702d80 x1875062187574272/t81604378829(81604378829) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/608 e 0 to 0 dl 1788201701 ref 2 fl Interpret:RQU/604/0 rc 301/301 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201685.004296:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201685.004298:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201685.004302:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201685.004333:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378829 00000100:00080000:2.0:1788201685.004351:0:2400:0:(client.c:3553:ptlrpc_replay_req()) @@@ REPLAY req@ffff8e82894d7480 x1875062187574528/t81604378830(81604378830) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201597 ref 2 fl New:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788201685.004359:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d702d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187574272:192.168.201.121@tcp:101:createmany.0 00000100:00100000:2.0:1788201685.004373:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187574528:192.168.201.121@tcp:35:createmany.0 00000100:00080000:2.0:1788201685.006494:0:2400:0:(client.c:3438:ptlrpc_replay_interpret()) @@@ got reply req@ffff8e82894d7480 x1875062187574528/t81604378830(81604378830) o35->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:23/10 lens 392/456 e 0 to 0 dl 1788201701 ref 3 fl Interpret:RQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788201685.006504:0:2400:0:(import.c:1700:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788201685.006507:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378724 gen 12 00000100:00100000:2.0:1788201685.006510:0:2400:0:(client.c:3169:ptlrpc_free_committed()) @@@ stopping search req@ffff8e828793c700 x1875062187537920/t81604378726(81604378726) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 528/448 e 6 to 0 dl 1788201702 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201685.006515:0:2400:0:(recover.c:53:ptlrpc_replay_next()) import ffff8e82871d2800 from lustre-MDT0000_UUID committed 81604378724 last 81604378830 00000100:00080000:2.0:1788201685.006529:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:2.0:1788201685.006897:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187574528:192.168.201.121@tcp:35:createmany.0 00010000:00080000:1.0:1788201685.006957:0:42631:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff8e8282a4b800 to lustre-MDT0000_UUID@192.168.201.121@tcp 00010000:00010000:1.0:1788201685.006962:0:42631:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff8e8282a4b800 (53) 00000080:00200000:1.0:1788201685.007100:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x109:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007155:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65e00/0x9ae3109cee9e23df lrc: 0/0,0 mode: --/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2eaf expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007176:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x108:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007201:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993800/0x9ae3109cee9e23d1 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2e7e expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007214:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x107:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007223:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b200/0x9ae3109cee9e23c3 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2e4d expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007233:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x106:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007247:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64000/0x9ae3109cee9e23b5 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2e1c expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007261:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x105:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007267:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0a00/0x9ae3109cee9e23a7 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x105:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2deb expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007276:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x104:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007282:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1000/0x9ae3109cee9e2399 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x104:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2dba expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007290:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x103:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007297:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558800/0x9ae3109cee9e238b lrc: 0/0,0 mode: --/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2d89 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007304:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x102:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007344:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65400/0x9ae3109cee9e237d lrc: 0/0,0 mode: --/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2d58 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007353:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x101:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007359:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64600/0x9ae3109cee9e236f lrc: 0/0,0 mode: --/PR res: [0x200006991:0x101:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2d27 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007367:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x100:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007374:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65000/0x9ae3109cee9e2361 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x100:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2cf6 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007381:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xff:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007387:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559400/0x9ae3109cee9e2353 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2cc5 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007398:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xfe:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007405:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558600/0x9ae3109cee9e2345 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2c94 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007413:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xfd:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007419:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558200/0x9ae3109cee9e2337 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2c63 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007427:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xfc:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007433:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64e00/0x9ae3109cee9e2329 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xfc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2c32 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007440:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xfb:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007446:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65200/0x9ae3109cee9e231b lrc: 0/0,0 mode: --/PR res: [0x200006991:0xfb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2c01 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007453:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xfa:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007458:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65c00/0x9ae3109cee9e230d lrc: 0/0,0 mode: --/PR res: [0x200006991:0xfa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2bd0 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007466:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xf9:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007480:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64800/0x9ae3109cee9e22ff lrc: 0/0,0 mode: --/PR res: [0x200006991:0xf9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2b9f expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007488:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xf8:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007495:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64200/0x9ae3109cee9e22f1 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xf8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2b6e expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007506:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xf7:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007512:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64c00/0x9ae3109cee9e22e3 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xf7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2b3d expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007520:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xf6:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007526:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f64a00/0x9ae3109cee9e22d5 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xf6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2b0c expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007534:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xf5:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007540:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65800/0x9ae3109cee9e22c7 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xf5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2adb expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007554:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xf4:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007561:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290f65600/0x9ae3109cee9e22b9 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xf4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2aaa expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007569:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xf3:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007575:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560600/0x9ae3109cee9e22ab lrc: 0/0,0 mode: --/PR res: [0x200006991:0xf3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2a79 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007583:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xf2:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007589:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560800/0x9ae3109cee9e229d lrc: 0/0,0 mode: --/PR res: [0x200006991:0xf2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2a48 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007597:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xf1:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007603:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558e00/0x9ae3109cee9e228f lrc: 0/0,0 mode: --/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2a17 expref: -99 pid: 42264 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007614:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x2000090a1:0x3:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007631:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560200/0x9ae3109cee9e2288 lrc: 0/0,0 mode: --/PR res: [0x2000090a1:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc29fb expref: -99 pid: 42260 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201685.007659:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561000/0x9ae3109cee9e2281 lrc: 0/0,0 mode: --/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x7cf2114380cc29df expref: -99 pid: 42260 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007672:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xf0:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007682:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559e00/0x9ae3109cee9e2273 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xf0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc29ae expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007695:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xef:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007705:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558a00/0x9ae3109cee9e2265 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc297d expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007715:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xee:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007721:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559200/0x9ae3109cee9e2257 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc294c expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007729:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xed:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007735:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558400/0x9ae3109cee9e2249 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc291b expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007742:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xec:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007748:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561200/0x9ae3109cee9e223b lrc: 0/0,0 mode: --/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc28ea expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007760:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xeb:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007766:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561c00/0x9ae3109cee9e222d lrc: 0/0,0 mode: --/PR res: [0x200006991:0xeb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc28b9 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007774:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xea:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007780:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1800/0x9ae3109cee9e221f lrc: 0/0,0 mode: --/PR res: [0x200006991:0xea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2888 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007787:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xe9:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007794:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee0800/0x9ae3109cee9e2211 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xe9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2857 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007801:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xe8:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007807:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561800/0x9ae3109cee9e2203 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xe8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2826 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007814:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xe7:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007820:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560a00/0x9ae3109cee9e21f5 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xe7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc27f5 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007827:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xe6:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007833:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560000/0x9ae3109cee9e21e7 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xe6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc27c4 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007840:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xe5:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007846:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560e00/0x9ae3109cee9e21d9 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xe5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2793 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007857:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xe4:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007863:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561a00/0x9ae3109cee9e21cb lrc: 0/0,0 mode: --/PR res: [0x200006991:0xe4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2762 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007870:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xe3:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007876:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561400/0x9ae3109cee9e21bd lrc: 0/0,0 mode: --/PR res: [0x200006991:0xe3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2731 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007887:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xe2:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007914:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289560c00/0x9ae3109cee9e21af lrc: 0/0,0 mode: --/PR res: [0x200006991:0xe2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2700 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007929:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xe1:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007939:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992e00/0x9ae3109cee9e21a1 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc26cf expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007953:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xe0:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007963:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992800/0x9ae3109cee9e2193 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc269e expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007976:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xdf:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.007986:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992400/0x9ae3109cee9e2185 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc266d expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.007998:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xde:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.008008:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992c00/0x9ae3109cee9e2177 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc263c expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.008020:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xdd:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.008027:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559600/0x9ae3109cee9e2169 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc260b expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.008035:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xdc:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.008041:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289558000/0x9ae3109cee9e215b lrc: 0/0,0 mode: --/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc25da expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.008048:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xdb:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.008055:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289561600/0x9ae3109cee9e214d lrc: 0/0,0 mode: --/PR res: [0x200006991:0xdb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc25a9 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.008063:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xda:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.008069:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559000/0x9ae3109cee9e213f lrc: 0/0,0 mode: --/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2578 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.008077:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xd9:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.008083:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289559800/0x9ae3109cee9e2131 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2547 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.008091:0:42631:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xd8:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.008096:0:42631:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993000/0x9ae3109cee9e2123 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000000 nid: local remote: 0x7cf2114380cc2516 expref: -99 pid: 42256 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.008102:0:42631:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff8e8282a4b800 00010000:00010000:1.0:1788201685.008497:0:42631:0:(ldlm_request.c:2860:replay_one_lock()) ### replaying lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc2ec4 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00080000:1.0:1788201685.008528:0:42631:0:(ldlm_request.c:2910:lock_can_replay()) check lock replay limit, inflights = 1(8) 00000100:00100000:3.0:1788201685.009157:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187589248:192.168.201.121@tcp:101:ldlm_lock_repla.0 00010000:00010000:3.0:1788201685.011844:0:2400:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc36b9 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000100:00080000:3.0:1788201685.011861:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00100000:3.0:1788201685.011892:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187589248:192.168.201.121@tcp:101:ldlm_lock_repla.0 00000100:00100000:3.0:1788201685.011920:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187589376:192.168.201.121@tcp:400:ptlrpcd_rcv.0 00000100:00100000:3.0:1788201685.019788:0:2400:0:(client.c:3125:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff8e8282a4b800: committing for last_committed 81604378830 gen 12 00000100:00080000:3.0:1788201685.020113:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:3.0:1788201685.020127:0:2400:0:(client.c:3247:ptlrpc_resend_req()) @@@ going to resend req@ffff8e8289538a80 x1875062187589376/t0(0) o400->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201701 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201685.020139:0:2400:0:(client.c:3256:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff8e8289538a80 x1875062187589376/t0(0) o400->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 0 dl 1788201701 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201685.020148:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:02000000:2.0:1788201685.023356:0:2400:0:(import.c:1743:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff8e8282a4b800: Connection restored to 192.168.201.121@tcp (at 192.168.201.121@tcp) 00000100:00080000:2.0:1788201685.023371:0:2400:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000ab5c9557): req@ffff8e828793dc00 x1875062187582976/t0(0) o41->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201685.023389:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187589376:192.168.201.121@tcp:400:ptlrpcd_rcv.0 00000100:00100000:0.0:1788201685.023602:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165180 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.027334:0:42607:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job lfs:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42607:1875062187582976:192.168.201.121@tcp:41:lfs.0 00800000:00000004:2.0:1788201685.027362:0:42607:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff8e8282a4b800: update ffff8e8291bdb7c0 cache blocks 317372/353529 objects 1023537/1024000 00000080:00200080:2.0:1788201685.027499:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.027516:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00800000:00000004:2.0:1788201685.027523:0:42607:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff8e8282a4b800: age 3033, max_age 3144 00000100:00100000:2.0:1788201685.027584:0:42607:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6d80 pname:cluuid:pid:xid:nid:opc:job lfs:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42607:1875062187589504:192.168.201.121@tcp:41:lfs.0 00000100:00100000:2.0:1788201685.027670:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.031946:0:42607:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6d80 pname:cluuid:pid:xid:nid:opc:job lfs:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42607:1875062187589504:192.168.201.121@tcp:41:lfs.0 00800000:00000004:2.0:1788201685.031976:0:42607:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff8e8282a4b800: update ffff8e82871ad178 cache blocks 317418/353529 objects 1023742/1024000 00000080:00200080:2.0:1788201685.032133:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032151:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032159:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032163:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032167:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032170:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032174:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032177:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032180:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032183:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032186:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032189:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032192:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032195:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032199:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032202:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032205:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032208:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032211:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032214:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032217:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032220:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032223:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032226:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032230:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032232:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032236:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032239:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032243:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032245:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032249:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032251:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032255:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032257:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032261:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032263:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032267:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032269:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032272:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032275:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032278:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032281:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032284:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032287:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032290:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032293:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032297:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032300:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032303:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032343:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032350:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032355:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032358:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032361:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032365:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032368:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032371:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032374:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032377:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032380:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032383:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032385:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032389:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032392:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032395:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032398:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032401:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032404:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032407:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032410:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032413:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032416:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032419:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032422:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032425:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032427:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032430:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032433:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032437:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032439:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032443:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032445:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032449:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032451:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032454:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032457:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032461:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032463:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032467:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032470:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032473:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032476:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032480:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032484:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032487:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032490:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032494:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032497:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032501:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032504:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032507:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032511:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032514:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032518:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032522:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032526:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032530:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032533:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032537:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032541:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032544:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032577:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032581:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032584:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032588:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032591:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032594:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032597:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032600:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032602:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032605:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032608:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032611:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032614:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032617:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00800000:00000080:2.0:1788201685.032630:0:42607:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.032634:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.032641:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00020000:00000004:2.0:1788201685.032661:0:42607:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff8e8282a4b800: age 3033, max_age 3144 00000100:00100000:2.0:1788201685.032699:0:42607:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6d80 pname:cluuid:pid:xid:nid:opc:job lfs:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42607:1875062187589632:192.168.201.121@tcp:13:lfs.0 00000100:00100000:2.0:1788201685.032762:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.034917:0:42607:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6d80 pname:cluuid:pid:xid:nid:opc:job lfs:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42607:1875062187589632:192.168.201.121@tcp:13:lfs.0 00020000:00000004:2.0:1788201685.034943:0:42607:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff8e8282a4b800: update ffff8e82871ab7c0 cache blocks 901369/958279 objects 261623/262144 00000080:00200080:2.0:1788201685.035149:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.035165:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00020000:00000004:2.0:1788201685.035173:0:42607:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff8e8282a4b800: age 3033, max_age 3144 00000100:00100000:2.0:1788201685.035205:0:42607:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6d80 pname:cluuid:pid:xid:nid:opc:job lfs:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42607:1875062187589760:192.168.201.121@tcp:13:lfs.0 00000100:00100000:2.0:1788201685.035272:0:42607:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.038729:0:42607:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6d80 pname:cluuid:pid:xid:nid:opc:job lfs:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42607:1875062187589760:192.168.201.121@tcp:13:lfs.0 00020000:00000004:2.0:1788201685.038760:0:42607:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff8e8282a4b800: update ffff8e82871a9e08 cache blocks 901369/958279 objects 261655/262144 00000080:00200080:2.0:1788201685.038814:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038831:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038838:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038842:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038846:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038849:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038852:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038855:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038858:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038861:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038864:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038868:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038872:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038876:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038879:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038882:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038886:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038889:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038893:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038896:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038900:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038903:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038907:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038910:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038914:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038918:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038922:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038925:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038929:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038932:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038936:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038940:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038943:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038947:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038950:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038954:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038957:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038961:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038965:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038968:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038972:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038976:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038980:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038983:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038986:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.038990:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.038993:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039009:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039014:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039017:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039021:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039024:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039028:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039032:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039035:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039038:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039042:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039046:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039050:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039054:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039058:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039061:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039066:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039069:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039073:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039077:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039081:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039084:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039088:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039092:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039096:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039099:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039103:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039107:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039111:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039114:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039118:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039121:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039125:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039129:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039132:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039136:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039140:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039144:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039148:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039151:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039155:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039159:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039163:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039166:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039170:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039173:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039177:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039179:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039182:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039184:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039186:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039188:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039190:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039192:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039193:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039195:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039197:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039199:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039201:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039202:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039204:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039206:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039208:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039209:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039211:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039213:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039216:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039218:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039220:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039222:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039224:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039225:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039227:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039229:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039231:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039232:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039234:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039236:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200080:2.0:1788201685.039238:0:42607:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988) cmd=c00866a4 arg=7ffe83b4ec50 00020000:00000080:2.0:1788201685.039240:0:42607:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff8e8282a4b800: cmd=c00866a4 len=592 karg=ffff8e8292318c00 uarg=0000000000000000 00000080:00200008:2.0:1788201685.039249:0:42607:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff8e8292531988), flags 100000 00000080:00200000:2.0:1788201685.132551:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.132578:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.132598:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc36b9 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788201685.132612:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc36b9 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788201685.132619:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.132698:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.132703:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e23e6 00000080:00010000:2.0:1788201685.132712:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23e6 from it ffffaa1801973c10 00010000:00010000:2.0:1788201685.132717:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc36b9 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788201685.132727:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc36b9 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788201685.132743:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.132754:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-0, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.132758:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-0, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.132769:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-0,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.132805:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2410 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.132810:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2410 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.132817:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2410 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.132836:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6d80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187589888:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.132928:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230d80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.140178:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6d80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187589888:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201685.140199:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2410 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc38f7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.140214:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2410 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc38f7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.140220:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2410 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc38f7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.140228:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2410 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc38f7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.140236:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2410 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc38f7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.140245:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2410 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc38f7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.140250:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2410 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc38f7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.140254:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2410 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc38f7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201685.140262:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d6d80 x1875062187589888/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201685.140269:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2410 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc38f7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201685.140277:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2410 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc38f7 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.140283:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/216 (0 0) 00000080:00200000:3.0:1788201685.140288:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201685.140416:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xd8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201685.140425:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201685.140458:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xd8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201685.140515:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xd8:0x0](ffff8e8289e5c2c8): rc = 0 00000080:00200000:3.0:1788201685.140529:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xd8:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201685.140536:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xd8:0x0](ffff8e8289e5c2c8) for lock 0x9ae3109cee9e2410 00000080:00010000:3.0:1788201685.140543:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xd8:0x0](ffff8e8289e5c2c8) 00000080:00010000:3.0:1788201685.140547:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2410 from it ffffaa1801973e00 00010000:00010000:3.0:1788201685.140551:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2410 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc38f7 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.140559:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2410 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc38f7 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.140589:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201685.140607:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201685.140615:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc36b9 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788201685.140637:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc36b9 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:3.0:1788201685.140651:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201685.140656:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201685.140660:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e23e6 00000080:00010000:3.0:1788201685.140664:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e23e6 from it ffffaa1801973d70 00010000:00010000:3.0:1788201685.140667:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc36b9 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788201685.140674:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc36b9 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:3.0:1788201685.140688:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201685.140695:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-0, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201685.140722:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc36b9 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788201685.140745:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2410 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc38f7 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.140752:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xd8:0x0]: losing layout lock 00010000:00010000:3.0:1788201685.140780:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2410 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc38f7 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.140787:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc36b9 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788201685.140793:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201685.140798:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2410 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xd8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc38f7 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.140812:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e23e6 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc36b9 expref: -99 pid: 42275 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000020:00100000:3.0:1788201685.140831:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201685.140840:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289538700 x1875062187590144/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201685.140863:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187590144:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201685.141856:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1240 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.155121:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187590144:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201685.155177:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xd8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201685.155189:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201685.155213:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xd8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201685.155221:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xd8:0x0](ffff8e8289e5c2c8) 00000080:00200000:3.0:1788201685.155909:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201685.155930:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201685.155983:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2417 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.155993:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2417 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.156004:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2417 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201685.156025:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953ad80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187590272:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201685.156110:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1000 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.162103:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953ad80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187590272:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201685.162131:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2417 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3921 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.162144:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2417 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3921 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.162153:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2417 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3921 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.162162:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2417 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3921 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.162172:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2417 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3921 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.162177:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2417 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3921 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.162181:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2417 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3921 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201685.162189:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828953ad80 x1875062187590272/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201685.162199:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2417 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3921 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.162208:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201685.162213:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201685.162235:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201685.162238:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2417 00000080:00010000:3.0:1788201685.162242:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201685.162244:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2417 from it ffffaa1801973c10 00010000:00010000:3.0:1788201685.162246:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2417 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3921 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.162255:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2417 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3921 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.162273:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201685.162286:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-1, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201685.162289:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-1, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201685.162332:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-1,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201685.162364:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e241e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.162369:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e241e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.162375:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e241e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201685.162389:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953ad80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187590400:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201685.162448:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1000 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.165412:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953ad80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187590400:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201685.165439:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e241e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc393d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.165455:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e241e lrc: 4/1,0 mode: --/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc393d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.165460:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e241e lrc: 4/1,0 mode: --/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc393d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.165468:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e241e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc393d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.165539:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e241e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc393d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.165550:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e241e lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc393d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.165558:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e241e lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc393d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.165565:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e241e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc393d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201685.165576:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828953ad80 x1875062187590400/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201685.165588:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e241e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc393d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201685.165602:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e241e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc393d expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.165610:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/217 (0 0) 00000080:00200000:3.0:1788201685.165616:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201685.165733:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xd9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201685.165745:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201685.165770:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xd9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201685.165821:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xd9:0x0](ffff8e8289e5d348): rc = 0 00000080:00200000:3.0:1788201685.165833:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xd9:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201685.165838:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xd9:0x0](ffff8e8289e5d348) for lock 0x9ae3109cee9e241e 00000080:00010000:3.0:1788201685.165846:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xd9:0x0](ffff8e8289e5d348) 00000080:00010000:3.0:1788201685.165849:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e241e from it ffffaa1801973e00 00010000:00010000:3.0:1788201685.165851:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e241e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc393d expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.165858:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e241e lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc393d expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.167047:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201685.167072:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201685.167084:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2417 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3921 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.167097:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2417 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3921 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.167103:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201685.167109:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201685.167112:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2417 00000080:00010000:3.0:1788201685.167117:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2417 from it ffffaa1801973d70 00010000:00010000:3.0:1788201685.167121:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2417 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3921 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.167133:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2417 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3921 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201685.167144:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201685.167151:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-1, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201685.167178:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2417 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3921 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201685.167203:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e241e lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc393d expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.167213:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xd9:0x0]: losing layout lock 00010000:00010000:3.0:1788201685.167246:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e241e lrc: 1/0,0 mode: --/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc393d expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.167253:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2417 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3921 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201685.167259:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201685.167262:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e241e lrc: 0/0,0 mode: --/PR res: [0x200006991:0xd9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc393d expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.167279:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2417 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3921 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201685.167296:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201685.169313:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828953ad80 x1875062187590656/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201685.169345:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953ad80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187590656:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.184380:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1240 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.184420:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953ad80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187590656:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.184667:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xd9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.184678:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.184720:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xd9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.184727:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xd9:0x0](ffff8e8289e5d348) 00000080:00200000:2.0:1788201685.184804:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.184821:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.184864:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2425 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.184871:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2425 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.184986:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2425 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.185008:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187590784:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.185085:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230000 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.187497:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187590784:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.187517:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2425 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3967 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.187526:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2425 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3967 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.187534:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2425 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3967 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.187542:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2425 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3967 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.187553:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2425 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3967 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.187561:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2425 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3967 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.187569:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2425 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3967 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.187580:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d4e00 x1875062187590784/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.187594:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2425 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3967 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.187612:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.187619:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.187779:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.187785:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2425 00000080:00010000:2.0:1788201685.187790:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.187792:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2425 from it ffffaa1801973c10 00010000:00010000:2.0:1788201685.187795:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2425 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3967 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.187801:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2425 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3967 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.187820:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.187841:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-2, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.187844:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-2, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.187862:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-2,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.188079:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462a00/0x9ae3109cee9e242c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.188087:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462a00/0x9ae3109cee9e242c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.188096:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462a00/0x9ae3109cee9e242c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.188117:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187590912:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.188233:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230000 going to sleep for 11 seconds 00010000:00010000:0.0:1788201685.189820:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201685.197429:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187590912:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:0.0:1788201685.197451:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462a00/0x9ae3109cee9e242c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3983 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201685.197466:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462a00/0x9ae3109cee9e242c lrc: 4/1,0 mode: --/PR res: [0x200006991:0xda:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3983 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201685.197473:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462a00/0x9ae3109cee9e242c lrc: 4/1,0 mode: --/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3983 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201685.197484:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462a00/0x9ae3109cee9e242c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3983 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201685.197495:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462a00/0x9ae3109cee9e242c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3983 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201685.197507:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462a00/0x9ae3109cee9e242c lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3983 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201685.197515:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462a00/0x9ae3109cee9e242c lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3983 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201685.197523:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462a00/0x9ae3109cee9e242c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3983 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788201685.197535:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d4e00 x1875062187590912/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788201685.197543:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462a00/0x9ae3109cee9e242c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3983 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201685.197552:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462a00/0x9ae3109cee9e242c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3983 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201685.197563:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/218 (0 0) 00000080:00200000:0.0:1788201685.197570:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201685.197671:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xda:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201685.197681:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201685.197717:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xda:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201685.197763:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xda:0x0](ffff8e8289e5db88): rc = 0 00000080:00200000:0.0:1788201685.197775:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xda:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201685.197779:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xda:0x0](ffff8e8289e5db88) for lock 0x9ae3109cee9e242c 00000080:00010000:0.0:1788201685.197787:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xda:0x0](ffff8e8289e5db88) 00000080:00010000:0.0:1788201685.197790:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e242c from it ffffaa1801973e00 00010000:00010000:0.0:1788201685.197792:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462a00/0x9ae3109cee9e242c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3983 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201685.197799:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462a00/0x9ae3109cee9e242c lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3983 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201685.197823:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201685.197835:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201685.197841:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2425 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3967 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201685.197849:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2425 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3967 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201685.197857:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201685.197861:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201685.197863:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2425 00000080:00010000:0.0:1788201685.197865:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2425 from it ffffaa1801973d70 00010000:00010000:0.0:1788201685.197867:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2425 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3967 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201685.197871:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2425 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3967 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201685.197875:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201685.197894:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-2, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201685.197919:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2425 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3967 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201685.197939:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462a00/0x9ae3109cee9e242c lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3983 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201685.197948:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xda:0x0]: losing layout lock 00010000:00010000:0.0:1788201685.197976:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462a00/0x9ae3109cee9e242c lrc: 1/0,0 mode: --/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3983 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201685.197985:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2425 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3967 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201685.197991:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201685.197994:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462a00/0x9ae3109cee9e242c lrc: 0/0,0 mode: --/PR res: [0x200006991:0xda:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3983 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201685.198010:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462400/0x9ae3109cee9e2425 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3967 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201685.198028:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201685.198035:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793d180 x1875062187591168/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201685.198056:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187591168:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201685.198123:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.207820:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187591168:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.207898:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xda:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.207910:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.207938:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xda:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.207945:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xda:0x0](ffff8e8289e5db88) 00000080:00200000:2.0:1788201685.208024:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.208045:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.208101:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463600/0x9ae3109cee9e2433 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.208110:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463600/0x9ae3109cee9e2433 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.208121:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463600/0x9ae3109cee9e2433 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.208139:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5c00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187591296:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.208215:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230000 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.211238:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5c00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187591296:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.211264:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463600/0x9ae3109cee9e2433 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39ad expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.211275:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463600/0x9ae3109cee9e2433 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39ad expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.211286:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463600/0x9ae3109cee9e2433 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc39ad expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.211296:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463600/0x9ae3109cee9e2433 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39ad expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.211334:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463600/0x9ae3109cee9e2433 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc39ad expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.211343:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463600/0x9ae3109cee9e2433 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc39ad expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.211349:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463600/0x9ae3109cee9e2433 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc39ad expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.211359:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d5c00 x1875062187591296/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.211373:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463600/0x9ae3109cee9e2433 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39ad expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.211384:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.211390:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.211415:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.211422:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2433 00000080:00010000:2.0:1788201685.211427:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.211431:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2433 from it ffffaa1801973c10 00010000:00010000:2.0:1788201685.211434:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463600/0x9ae3109cee9e2433 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc39ad expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.211444:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463600/0x9ae3109cee9e2433 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39ad expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.211466:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.211670:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.211675:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.211696:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.211735:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463400/0x9ae3109cee9e243a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.211742:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463400/0x9ae3109cee9e243a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.211750:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463400/0x9ae3109cee9e243a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.211771:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5c00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187591424:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.211855:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230000 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.218675:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5c00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187591424:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201685.218706:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463400/0x9ae3109cee9e243a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39c9 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.218723:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463400/0x9ae3109cee9e243a lrc: 4/1,0 mode: --/PR res: [0x200006991:0xdb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39c9 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.218730:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463400/0x9ae3109cee9e243a lrc: 4/1,0 mode: --/PR res: [0x200006991:0xdb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39c9 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.218739:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463400/0x9ae3109cee9e243a lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xdb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc39c9 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.218746:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463400/0x9ae3109cee9e243a lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xdb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39c9 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.218755:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463400/0x9ae3109cee9e243a lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xdb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc39c9 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.218760:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463400/0x9ae3109cee9e243a lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xdb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc39c9 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.218765:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463400/0x9ae3109cee9e243a lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xdb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc39c9 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201685.218778:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d5c00 x1875062187591424/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201685.218787:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463400/0x9ae3109cee9e243a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39c9 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201685.218796:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463400/0x9ae3109cee9e243a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39c9 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.218803:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/219 (0 0) 00000080:00200000:3.0:1788201685.218808:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201685.218843:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xdb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201685.218854:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201685.218880:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xdb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201685.218940:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xdb:0x0](ffff8e8289e43248): rc = 0 00000080:00200000:3.0:1788201685.218953:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xdb:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201685.218959:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xdb:0x0](ffff8e8289e43248) for lock 0x9ae3109cee9e243a 00000080:00010000:3.0:1788201685.218968:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xdb:0x0](ffff8e8289e43248) 00000080:00010000:3.0:1788201685.218972:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e243a from it ffffaa1801973e00 00010000:00010000:3.0:1788201685.218975:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463400/0x9ae3109cee9e243a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc39c9 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.218983:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463400/0x9ae3109cee9e243a lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xdb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc39c9 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.219012:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201685.219028:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201685.219038:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463600/0x9ae3109cee9e2433 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39ad expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.219049:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463600/0x9ae3109cee9e2433 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39ad expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.219055:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201685.219060:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201685.219063:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2433 00000080:00010000:3.0:1788201685.219067:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2433 from it ffffaa1801973d70 00010000:00010000:3.0:1788201685.219070:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463600/0x9ae3109cee9e2433 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc39ad expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.219078:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463600/0x9ae3109cee9e2433 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39ad expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201685.219087:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201685.219093:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201685.219114:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463600/0x9ae3109cee9e2433 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc39ad expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201685.219130:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463400/0x9ae3109cee9e243a lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xdb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc39c9 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.219136:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xdb:0x0]: losing layout lock 00010000:00010000:3.0:1788201685.219158:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463400/0x9ae3109cee9e243a lrc: 1/0,0 mode: --/PR res: [0x200006991:0xdb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc39c9 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.219163:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463600/0x9ae3109cee9e2433 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc39ad expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201685.219167:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201685.219169:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463400/0x9ae3109cee9e243a lrc: 0/0,0 mode: --/PR res: [0x200006991:0xdb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc39c9 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.219179:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463600/0x9ae3109cee9e2433 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc39ad expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201685.219191:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201685.219196:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289539c00 x1875062187591680/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201685.219212:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539c00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187591680:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201685.219284:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1480 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.232873:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539c00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187591680:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201685.232927:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xdb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201685.232939:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201685.232966:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xdb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201685.232975:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xdb:0x0](ffff8e8289e43248) 00000080:00200000:3.0:1788201685.233053:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201685.233072:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201685.233124:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2441 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.233133:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2441 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.233142:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2441 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201685.233162:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187591808:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201685.233234:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1d80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.234986:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187591808:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201685.235008:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2441 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39f3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.235019:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2441 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39f3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.235031:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2441 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc39f3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.235041:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2441 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39f3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.235052:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2441 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc39f3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.235060:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2441 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc39f3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.235067:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2441 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc39f3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201685.235079:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289538380 x1875062187591808/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201685.235098:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2441 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39f3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.235110:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201685.235115:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201685.235138:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201685.235145:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2441 00000080:00010000:3.0:1788201685.235150:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201685.235153:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2441 from it ffffaa1801973c10 00010000:00010000:3.0:1788201685.235157:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2441 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc39f3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.235165:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2441 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39f3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.235187:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201685.235207:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-4, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201685.235211:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-4, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201685.235230:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-4,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201685.235260:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e2448 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.235267:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e2448 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.235275:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e2448 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201685.235295:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187591936:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201685.235371:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1d80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.245136:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187591936:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201685.245167:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e2448 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a0f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.245189:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e2448 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a0f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.245197:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e2448 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a0f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.245208:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e2448 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a0f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.245218:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e2448 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a0f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.245231:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e2448 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a0f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.245239:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e2448 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a0f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.245246:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e2448 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a0f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201685.245261:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289538380 x1875062187591936/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201685.245272:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e2448 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a0f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201685.245285:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e2448 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a0f expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.245294:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/220 (0 0) 00000080:00200000:1.0:1788201685.245301:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201685.248848:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xdc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201685.248864:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201685.248900:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xdc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201685.248955:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xdc:0x0](ffff8e8289e59988): rc = 0 00000080:00200000:1.0:1788201685.248971:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xdc:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:1.0:1788201685.248978:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xdc:0x0](ffff8e8289e59988) for lock 0x9ae3109cee9e2448 00000080:00010000:1.0:1788201685.248988:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xdc:0x0](ffff8e8289e59988) 00000080:00010000:1.0:1788201685.248991:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2448 from it ffffaa1801973e00 00010000:00010000:1.0:1788201685.248997:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e2448 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3a0f expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.249009:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e2448 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3a0f expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.249043:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201685.249061:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201685.249074:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2441 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39f3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.249084:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2441 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39f3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.249092:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201685.249096:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201685.249099:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2441 00000080:00010000:1.0:1788201685.249103:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2441 from it ffffaa1801973d70 00010000:00010000:1.0:1788201685.249105:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2441 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc39f3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.249113:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2441 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc39f3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201685.249121:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201685.249127:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-4, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201685.249150:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2441 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc39f3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201685.249172:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e2448 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3a0f expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.249182:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xdc:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.249212:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e2448 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3a0f expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.249220:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2441 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc39f3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201685.249227:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201685.249229:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b400/0x9ae3109cee9e2448 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xdc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3a0f expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.249247:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2441 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc39f3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201685.249266:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201685.249277:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d721c00 x1875062187592192/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201685.249299:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721c00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187592192:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201685.249525:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64a6c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788201685.276763:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721c00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187592192:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.278172:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xdc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.278186:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.278215:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xdc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.278223:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xdc:0x0](ffff8e8289e59988) 00000080:00200000:2.0:1788201685.278299:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.278348:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.278404:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e244f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.278415:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e244f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.278424:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e244f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.278447:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187592320:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.278537:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.301514:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187592320:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.301538:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e244f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a39 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.301551:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e244f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a39 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.301561:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e244f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a39 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.301571:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e244f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a39 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.301583:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e244f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a39 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.301594:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e244f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a39 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.301601:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e244f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a39 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.301611:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d4380 x1875062187592320/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.301862:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e244f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a39 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.301876:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.302121:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.302147:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.302154:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e244f 00000080:00010000:2.0:1788201685.302158:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.302162:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e244f from it ffffaa1801973c10 00010000:00010000:2.0:1788201685.302166:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e244f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a39 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.302173:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e244f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a39 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.302196:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.302217:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-5, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.302221:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-5, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.305879:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-5,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.305952:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2456 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.305963:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2456 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.305974:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2456 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.305999:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187592448:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.306134:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.314787:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187592448:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.314814:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2456 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a55 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.314835:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2456 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a55 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.314843:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2456 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a55 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.314854:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2456 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a55 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.314864:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2456 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a55 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.314875:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2456 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a55 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.314883:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2456 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a55 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.314890:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2456 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a55 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.314902:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d4380 x1875062187592448/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.314912:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2456 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a55 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.314923:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2456 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a55 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.314933:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/221 (0 0) 00000080:00200000:2.0:1788201685.314939:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.315006:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xdd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.315017:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.315052:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xdd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.315110:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xdd:0x0](ffff8e8289e5cb08): rc = 0 00000080:00200000:2.0:1788201685.315127:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xdd:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.315132:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xdd:0x0](ffff8e8289e5cb08) for lock 0x9ae3109cee9e2456 00000080:00010000:2.0:1788201685.315142:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xdd:0x0](ffff8e8289e5cb08) 00000080:00010000:2.0:1788201685.315145:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2456 from it ffffaa1801973e00 00010000:00010000:2.0:1788201685.315149:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2456 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3a55 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.315162:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2456 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3a55 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.315192:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.315208:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.315217:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e244f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a39 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.315226:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e244f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a39 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.315233:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.315238:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.315241:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e244f 00000080:00010000:2.0:1788201685.315245:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e244f from it ffffaa1801973d70 00010000:00010000:2.0:1788201685.315248:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e244f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a39 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.315254:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e244f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a39 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.315262:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.315268:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-5, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.315293:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e244f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3a39 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.316038:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2456 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3a55 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.316058:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xdd:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.316094:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2456 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3a55 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.316101:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e244f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3a39 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.316107:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.316110:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2456 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xdd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3a55 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.316128:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e244f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3a39 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.316146:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.316154:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894d4380 x1875062187592704/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.316173:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187592704:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.316282:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82922306c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.323951:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4380 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187592704:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.324016:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xdd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.324030:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.324058:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xdd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.324067:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xdd:0x0](ffff8e8289e5cb08) 00000080:00200000:2.0:1788201685.324155:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.324178:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.324233:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e245d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.324243:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e245d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.324251:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e245d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.324270:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6a00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187592832:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.324402:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.327111:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6a00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187592832:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.327138:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e245d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a7f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.327151:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e245d lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a7f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.327162:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e245d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a7f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.327173:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e245d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a7f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.327184:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e245d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a7f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.327192:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e245d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a7f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.327201:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e245d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a7f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.327212:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d6a00 x1875062187592832/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.327226:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e245d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a7f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.327238:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.327245:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.327272:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.327280:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e245d 00000080:00010000:2.0:1788201685.327285:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.327288:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e245d from it ffffaa1801973c10 00010000:00010000:2.0:1788201685.327291:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e245d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a7f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.327299:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e245d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a7f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.327442:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.327470:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-6, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.327474:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-6, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.327495:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-6,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.327534:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2464 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.327542:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2464 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.327551:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2464 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.327570:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6a00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187592960:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.327756:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.331269:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6a00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187592960:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.331296:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2464 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a9b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.331786:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2464 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xde:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a9b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.331796:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2464 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a9b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.331807:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2464 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a9b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.331817:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2464 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a9b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.331830:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2464 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a9b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.331843:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2464 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a9b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.331849:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2464 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a9b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.331862:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d6a00 x1875062187592960/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.331872:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2464 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a9b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.332078:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2464 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a9b expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.332091:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/222 (0 0) 00000080:00200000:2.0:1788201685.332098:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.332142:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xde:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.332154:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.332180:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xde:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.332236:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xde:0x0](ffff8e8289c14b08): rc = 0 00000080:00200000:2.0:1788201685.332250:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xde:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.332256:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xde:0x0](ffff8e8289c14b08) for lock 0x9ae3109cee9e2464 00000080:00010000:2.0:1788201685.332264:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xde:0x0](ffff8e8289c14b08) 00000080:00010000:2.0:1788201685.332268:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2464 from it ffffaa1801973e00 00010000:00010000:2.0:1788201685.332271:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2464 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3a9b expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.332285:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2464 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3a9b expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.332390:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.332408:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.332418:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e245d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a7f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.332429:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e245d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a7f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.332435:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.332439:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.332441:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e245d 00000080:00010000:2.0:1788201685.332444:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e245d from it ffffaa1801973d70 00010000:00010000:2.0:1788201685.332447:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e245d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3a7f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.332454:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e245d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3a7f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.332463:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.332470:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-6, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.332496:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e245d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3a7f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.332518:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2464 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3a9b expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.332528:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xde:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.332556:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2464 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3a9b expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.332563:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e245d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3a7f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.332568:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.332571:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2464 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xde:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3a9b expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.332585:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e245d lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3a7f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.332601:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.332609:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894d6a00 x1875062187593216/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.333397:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6a00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187593216:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.333496:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82922306c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.342162:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6a00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187593216:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.342225:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xde:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.342238:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.342266:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xde:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.342274:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xde:0x0](ffff8e8289c14b08) 00000080:00200000:2.0:1788201685.342482:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.342508:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.342563:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e246b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.342573:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e246b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.342584:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e246b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.342602:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4000 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187593344:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.342694:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.345912:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4000 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187593344:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.345937:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e246b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ac5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.345948:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e246b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ac5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.345958:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e246b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3ac5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.345969:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e246b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ac5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.345981:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e246b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3ac5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.345988:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e246b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3ac5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.345996:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e246b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3ac5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.346008:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d4000 x1875062187593344/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.346020:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e246b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ac5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.346034:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.346040:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.346064:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.346072:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e246b 00000080:00010000:2.0:1788201685.346077:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.346080:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e246b from it ffffaa1801973c10 00010000:00010000:2.0:1788201685.346083:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e246b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3ac5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.346092:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e246b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ac5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.346114:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.346143:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-7, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.346147:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-7, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.346167:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-7,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.346203:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2472 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.346212:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2472 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.346221:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2472 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.346238:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4000 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187593472:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.346344:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.350916:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4000 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187593472:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.350940:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2472 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ae1 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.350957:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2472 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ae1 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.350963:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2472 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ae1 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.350971:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2472 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3ae1 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.350982:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2472 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ae1 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.350994:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2472 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3ae1 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.351008:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2472 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3ae1 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.351016:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2472 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3ae1 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.351028:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d4000 x1875062187593472/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.351040:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2472 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ae1 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.351052:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2472 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ae1 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.351062:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/223 (0 0) 00000080:00200000:2.0:1788201685.351068:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.351108:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xdf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.351119:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.351148:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xdf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.351206:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xdf:0x0](ffff8e8289c13a88): rc = 0 00000080:00200000:2.0:1788201685.351223:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xdf:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.351229:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xdf:0x0](ffff8e8289c13a88) for lock 0x9ae3109cee9e2472 00000080:00010000:2.0:1788201685.351238:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xdf:0x0](ffff8e8289c13a88) 00000080:00010000:2.0:1788201685.351241:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2472 from it ffffaa1801973e00 00010000:00010000:2.0:1788201685.351245:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2472 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3ae1 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.351258:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2472 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3ae1 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.351290:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.351550:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.351561:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e246b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ac5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.351572:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e246b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ac5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.351578:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.351583:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.351586:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e246b 00000080:00010000:2.0:1788201685.351590:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e246b from it ffffaa1801973d70 00010000:00010000:2.0:1788201685.351593:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e246b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3ac5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.351599:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e246b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ac5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.351608:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.351615:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-7, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.352464:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e246b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3ac5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.352494:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2472 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3ae1 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.352503:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xdf:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.352535:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2472 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3ae1 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.352542:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e246b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3ac5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.352548:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.352551:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e2472 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xdf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3ae1 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.352569:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e246b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3ac5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.352585:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.352596:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894d4000 x1875062187593728/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.352797:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4000 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187593728:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.353009:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82922306c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.356983:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4000 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187593728:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.357040:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xdf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.357053:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.357080:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xdf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.357088:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xdf:0x0](ffff8e8289c13a88) 00000080:00200000:2.0:1788201685.357438:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.357457:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.357509:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2479 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.357519:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2479 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.357529:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2479 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.357550:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187593856:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.357637:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.360002:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187593856:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.360028:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2479 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b0b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.360039:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2479 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b0b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.360050:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2479 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b0b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.360061:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2479 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b0b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.360073:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2479 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b0b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.360082:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2479 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b0b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.360091:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2479 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b0b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.360103:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d6680 x1875062187593856/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.360117:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2479 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b0b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.360130:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.360137:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.360160:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.360165:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2479 00000080:00010000:2.0:1788201685.360170:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.360173:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2479 from it ffffaa1801973c10 00010000:00010000:2.0:1788201685.360176:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2479 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b0b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.360182:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2479 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b0b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.360200:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.360216:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-8, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.360219:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-8, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.360233:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-8,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.360258:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2480 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.360263:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2480 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.360269:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2480 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.360280:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187593984:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.360360:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.362998:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187593984:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.363024:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2480 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b27 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.363044:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2480 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b27 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.363052:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2480 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b27 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.363063:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2480 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b27 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.363072:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2480 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b27 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.363084:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2480 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b27 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.363097:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2480 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b27 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.363112:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2480 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b27 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.363121:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d6680 x1875062187593984/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.363128:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2480 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b27 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.363137:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2480 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b27 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.363144:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/224 (0 0) 00000080:00200000:2.0:1788201685.363149:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.363183:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xe0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.363193:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.363214:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xe0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.363258:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xe0:0x0](ffff8e8289c121c8): rc = 0 00000080:00200000:2.0:1788201685.363267:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xe0:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.363271:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xe0:0x0](ffff8e8289c121c8) for lock 0x9ae3109cee9e2480 00000080:00010000:2.0:1788201685.363278:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xe0:0x0](ffff8e8289c121c8) 00000080:00010000:2.0:1788201685.363281:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2480 from it ffffaa1801973e00 00010000:00010000:2.0:1788201685.363283:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2480 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3b27 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.363293:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2480 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3b27 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.363343:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.363357:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.363363:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2479 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b0b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.363370:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2479 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b0b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.363374:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.363376:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.363378:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2479 00000080:00010000:2.0:1788201685.363381:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2479 from it ffffaa1801973d70 00010000:00010000:2.0:1788201685.363383:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2479 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b0b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.363387:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2479 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b0b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.363392:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.363397:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-8, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.363415:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2479 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3b0b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.363431:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2480 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3b27 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.363437:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xe0:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.363458:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2480 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3b27 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.363463:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2479 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3b0b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.363475:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.363477:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2480 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xe0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3b27 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.363488:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e2479 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3b0b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.363501:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.363506:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894d6680 x1875062187594240/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.363523:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187594240:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.363590:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82922306c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.368071:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187594240:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.368132:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xe0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.368146:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.368176:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xe0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.368184:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xe0:0x0](ffff8e8289c121c8) 00000080:00200000:2.0:1788201685.368262:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.368279:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.371807:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e2487 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.371822:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e2487 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.371834:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e2487 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.371855:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187594368:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.373126:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.375373:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187594368:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.375403:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e2487 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b51 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.375415:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e2487 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b51 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.375428:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e2487 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b51 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.375438:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e2487 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b51 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.375451:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e2487 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b51 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.375460:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e2487 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b51 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.375468:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e2487 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b51 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.375482:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d6300 x1875062187594368/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.375497:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e2487 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b51 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.375509:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.375516:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.375543:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.375550:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2487 00000080:00010000:2.0:1788201685.375556:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.375559:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2487 from it ffffaa1801973c10 00010000:00010000:2.0:1788201685.375562:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e2487 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b51 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.375571:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e2487 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b51 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.375595:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.375627:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-9, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.375632:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-9, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.375659:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-9,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.375699:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e248e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.375705:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e248e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.375712:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e248e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.375726:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187594496:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.375799:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.379541:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187594496:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.379567:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e248e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b6d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.379589:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e248e lrc: 4/1,0 mode: --/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b6d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.379595:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e248e lrc: 4/1,0 mode: --/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b6d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.379603:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e248e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b6d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.379610:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e248e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b6d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.379693:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e248e lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b6d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.379704:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e248e lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b6d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.379708:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e248e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b6d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.379716:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d6300 x1875062187594496/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.379724:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e248e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b6d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.379734:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e248e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b6d expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.379741:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/225 (0 0) 00000080:00200000:2.0:1788201685.379746:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.379781:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xe1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.379789:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.379810:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xe1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.379855:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xe1:0x0](ffff8e8289c17448): rc = 0 00000080:00200000:2.0:1788201685.379865:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xe1:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.379870:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xe1:0x0](ffff8e8289c17448) for lock 0x9ae3109cee9e248e 00000080:00010000:2.0:1788201685.380506:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xe1:0x0](ffff8e8289c17448) 00000080:00010000:2.0:1788201685.380512:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e248e from it ffffaa1801973e00 00010000:00010000:2.0:1788201685.380517:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e248e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3b6d expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.380531:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e248e lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3b6d expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.380562:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201685.381377:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201685.381393:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e2487 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b51 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.381402:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e2487 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b51 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.381407:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201685.381412:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201685.381415:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2487 00000080:00010000:3.0:1788201685.381419:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2487 from it ffffaa1801973d70 00010000:00010000:3.0:1788201685.381421:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e2487 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b51 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.381427:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e2487 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b51 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201685.381434:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201685.381440:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-9, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201685.381463:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e2487 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3b51 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201685.381488:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e248e lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3b6d expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.381503:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xe1:0x0]: losing layout lock 00010000:00010000:3.0:1788201685.381537:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e248e lrc: 1/0,0 mode: --/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3b6d expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.381545:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e2487 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3b51 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201685.381551:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201685.381553:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463a00/0x9ae3109cee9e248e lrc: 0/0,0 mode: --/PR res: [0x200006991:0xe1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3b6d expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.381572:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462600/0x9ae3109cee9e2487 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3b51 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201685.381592:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201685.381602:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828953aa00 x1875062187594752/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201685.381635:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953aa00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187594752:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201685.381751:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1480 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.386829:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953aa00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187594752:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201685.386900:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xe1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201685.386916:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201685.386943:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xe1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201685.386950:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xe1:0x0](ffff8e8289c17448) 00000080:00200000:3.0:1788201685.387040:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201685.387060:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201685.387118:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2495 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.387128:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2495 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.387137:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2495 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201685.387157:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538000 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187594880:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201685.387263:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1480 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.391179:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538000 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187594880:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201685.391205:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2495 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b97 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.391213:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2495 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b97 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.392201:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2495 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b97 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.392220:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2495 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b97 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.392235:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2495 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b97 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.392243:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2495 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b97 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.392251:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2495 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b97 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201685.392265:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289538000 x1875062187594880/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201685.392278:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2495 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b97 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.392290:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201685.392297:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201685.392352:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201685.392359:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2495 00000080:00010000:3.0:1788201685.392364:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201685.392367:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2495 from it ffffaa1801973c10 00010000:00010000:3.0:1788201685.392370:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2495 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b97 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.392379:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2495 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b97 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.392403:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201685.392437:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-10, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201685.392441:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-10, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201685.392461:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-10,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201685.392499:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e249c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.392507:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e249c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.392516:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e249c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201685.392536:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538000 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187595008:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201685.392731:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1480 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.412025:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538000 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187595008:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201685.412054:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e249c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3bb3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.412079:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e249c lrc: 4/1,0 mode: --/PR res: [0x200006991:0xe2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3bb3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.412088:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e249c lrc: 4/1,0 mode: --/PR res: [0x200006991:0xe2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3bb3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.412098:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e249c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3bb3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.412110:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e249c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3bb3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.412122:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e249c lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xe2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3bb3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.412135:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e249c lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xe2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3bb3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.412143:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e249c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3bb3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201685.412156:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289538000 x1875062187595008/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201685.412167:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e249c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3bb3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201685.412181:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e249c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3bb3 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.412192:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/226 (0 0) 00000080:00200000:3.0:1788201685.412199:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201685.412241:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xe2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201685.412253:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201685.412278:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xe2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201685.412387:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xe2:0x0](ffff8e8289e42a08): rc = 0 00000080:00200000:3.0:1788201685.412401:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xe2:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201685.412407:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xe2:0x0](ffff8e8289e42a08) for lock 0x9ae3109cee9e249c 00000080:00010000:3.0:1788201685.412417:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xe2:0x0](ffff8e8289e42a08) 00000080:00010000:3.0:1788201685.412420:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e249c from it ffffaa1801973e00 00010000:00010000:3.0:1788201685.412424:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e249c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3bb3 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.412438:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e249c lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3bb3 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.412466:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201685.412483:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201685.412492:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2495 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b97 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.412501:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2495 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b97 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.412507:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201685.412511:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201685.412514:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2495 00000080:00010000:3.0:1788201685.412517:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2495 from it ffffaa1801973d70 00010000:00010000:3.0:1788201685.412520:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2495 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3b97 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.412527:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2495 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3b97 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201685.412534:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201685.412540:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-10, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201685.412560:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2495 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3b97 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201685.412578:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e249c lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3bb3 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.412587:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xe2:0x0]: losing layout lock 00010000:00010000:3.0:1788201685.412613:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e249c lrc: 1/0,0 mode: --/PR res: [0x200006991:0xe2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3bb3 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.412631:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2495 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3b97 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201685.412637:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201685.412648:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e249c lrc: 0/0,0 mode: --/PR res: [0x200006991:0xe2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3bb3 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.412662:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2495 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3b97 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201685.412678:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201685.412685:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289538000 x1875062187595264/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201685.412709:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538000 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187595264:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201685.412799:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1d80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.417447:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538000 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187595264:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201685.417507:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xe2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201685.417520:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201685.417550:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xe2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201685.417558:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xe2:0x0](ffff8e8289e42a08) 00000080:00200000:2.0:1788201685.419204:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.419231:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.419292:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24a3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.419303:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24a3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.419787:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24a3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.419814:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187595392:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.420041:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82922306c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.430966:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187595392:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.430993:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24a3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3bdd expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.431006:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24a3 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3bdd expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.431018:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24a3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3bdd expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.431034:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24a3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3bdd expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.431046:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24a3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3bdd expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.431055:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24a3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3bdd expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.431062:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24a3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3bdd expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.431074:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d6300 x1875062187595392/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.431090:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24a3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3bdd expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.431111:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.431121:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.431147:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.431154:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e24a3 00000080:00010000:2.0:1788201685.431160:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.431163:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24a3 from it ffffaa1801973c10 00010000:00010000:2.0:1788201685.431166:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24a3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3bdd expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.431176:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24a3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3bdd expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.431194:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.431211:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-11, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.431215:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-11, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.431231:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-11,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.431261:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e24aa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.431266:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e24aa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.431272:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e24aa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.431286:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187595520:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.431388:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82922306c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.435757:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187595520:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.435780:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e24aa lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3bf9 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.435796:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e24aa lrc: 4/1,0 mode: --/PR res: [0x200006991:0xe3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3bf9 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.435802:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e24aa lrc: 4/1,0 mode: --/PR res: [0x200006991:0xe3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3bf9 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.435811:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e24aa lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3bf9 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.435818:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e24aa lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3bf9 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.435827:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e24aa lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xe3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3bf9 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.435832:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e24aa lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xe3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3bf9 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.435836:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e24aa lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3bf9 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.435844:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d6300 x1875062187595520/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.435851:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e24aa lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3bf9 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.435860:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e24aa lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3bf9 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.435866:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/227 (0 0) 00000080:00200000:2.0:1788201685.435871:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.435909:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xe3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.435931:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.435954:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xe3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.436011:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xe3:0x0](ffff8e8289e400c8): rc = 0 00000080:00200000:2.0:1788201685.436028:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xe3:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.436033:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xe3:0x0](ffff8e8289e400c8) for lock 0x9ae3109cee9e24aa 00000080:00010000:2.0:1788201685.436042:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xe3:0x0](ffff8e8289e400c8) 00000080:00010000:2.0:1788201685.436046:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24aa from it ffffaa1801973e00 00010000:00010000:2.0:1788201685.436049:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e24aa lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3bf9 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.436057:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e24aa lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3bf9 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.436087:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.436102:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.436111:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24a3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3bdd expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.436121:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24a3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3bdd expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.436128:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.436132:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.436135:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e24a3 00000080:00010000:2.0:1788201685.436139:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24a3 from it ffffaa1801973d70 00010000:00010000:2.0:1788201685.436142:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24a3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3bdd expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.436154:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24a3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3bdd expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.436160:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.436166:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-11, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.436186:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24a3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3bdd expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.436201:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e24aa lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3bf9 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.436207:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xe3:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.436229:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e24aa lrc: 1/0,0 mode: --/PR res: [0x200006991:0xe3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3bf9 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.436234:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24a3 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3bdd expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.436238:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.436240:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e24aa lrc: 0/0,0 mode: --/PR res: [0x200006991:0xe3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3bf9 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.436250:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24a3 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3bdd expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.436262:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.436267:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894d6300 x1875062187595776/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.436284:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187595776:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.436393:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230180 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.440606:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187595776:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.440684:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xe3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.440697:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.440727:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xe3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.440735:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xe3:0x0](ffff8e8289e400c8) 00000080:00200000:2.0:1788201685.440820:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.440838:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.440889:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e24b1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.440897:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e24b1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.440904:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e24b1 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.440925:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187595904:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.440996:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82922306c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.444494:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187595904:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.444522:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e24b1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c23 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.444535:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e24b1 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c23 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.444552:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e24b1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3c23 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.444564:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e24b1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c23 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.444576:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e24b1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3c23 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.444585:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e24b1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3c23 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.444592:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e24b1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3c23 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.444604:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d5500 x1875062187595904/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.444690:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e24b1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c23 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.444706:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.444713:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.444741:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.444748:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e24b1 00000080:00010000:2.0:1788201685.444753:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.444756:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24b1 from it ffffaa1801973c10 00010000:00010000:2.0:1788201685.444760:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e24b1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3c23 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.444768:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e24b1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c23 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.444792:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.444818:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-12, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.444823:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-12, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.444844:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-12,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.444898:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e24b8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.444905:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e24b8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.444914:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e24b8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.445005:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187596032:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.445090:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82922306c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.451343:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187596032:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.451364:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e24b8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c3f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.451381:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e24b8 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xe4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c3f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.451391:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e24b8 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xe4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c3f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.451398:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e24b8 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3c3f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.451404:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e24b8 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c3f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.451413:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e24b8 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xe4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3c3f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.451418:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e24b8 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xe4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3c3f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.451423:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e24b8 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3c3f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.451431:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d5500 x1875062187596032/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.451438:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e24b8 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c3f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.451446:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e24b8 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c3f expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.451456:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/228 (0 0) 00000080:00200000:2.0:1788201685.451461:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.451502:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xe4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.451514:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.451540:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xe4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.451596:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xe4:0x0](ffff8e8289e44b08): rc = 0 00000080:00200000:2.0:1788201685.451609:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xe4:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.451614:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xe4:0x0](ffff8e8289e44b08) for lock 0x9ae3109cee9e24b8 00000080:00010000:2.0:1788201685.451635:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xe4:0x0](ffff8e8289e44b08) 00000080:00010000:2.0:1788201685.451646:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24b8 from it ffffaa1801973e00 00010000:00010000:2.0:1788201685.451650:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e24b8 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3c3f expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.451659:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e24b8 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3c3f expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.451689:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.451704:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.451711:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e24b1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c23 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.451718:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e24b1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c23 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.451722:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.451725:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.451728:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e24b1 00000080:00010000:2.0:1788201685.451730:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24b1 from it ffffaa1801973d70 00010000:00010000:2.0:1788201685.451732:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e24b1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3c23 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.451736:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e24b1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c23 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.451741:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.451746:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-12, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.451764:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e24b1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3c23 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.451780:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e24b8 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3c3f expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.451785:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xe4:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.451807:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e24b8 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xe4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3c3f expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.451812:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e24b1 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3c23 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.451816:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.451818:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e24b8 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xe4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3c3f expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.451827:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e24b1 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3c23 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.451843:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.451848:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894d5500 x1875062187596288/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.451864:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187596288:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.452065:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230180 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.456818:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187596288:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:0.0:1788201685.456957:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xe4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201685.457211:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201685.457242:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xe4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201685.457251:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xe4:0x0](ffff8e8289e44b08) 00000080:00200000:0.0:1788201685.457407:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201685.457427:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201685.457600:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e24bf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201685.457609:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e24bf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201685.457690:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e24bf lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201685.457713:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187596416:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201685.457846:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165300 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.460865:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187596416:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.460892:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e24bf lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c69 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.460905:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e24bf lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c69 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.460915:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e24bf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3c69 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.460926:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e24bf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c69 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.460937:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e24bf lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3c69 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.460945:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e24bf lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3c69 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.460957:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e24bf lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3c69 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.461104:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793e680 x1875062187596416/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.461119:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e24bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c69 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.461132:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.461139:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.461168:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.461175:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e24bf 00000080:00010000:2.0:1788201685.461181:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.461184:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24bf from it ffffaa1801973c10 00010000:00010000:2.0:1788201685.461187:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e24bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3c69 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.461196:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e24bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c69 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.461219:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.461240:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-13, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.461244:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-13, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.461264:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-13,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.461425:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24c6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.461433:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24c6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.461442:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24c6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.461462:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187596544:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.461648:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230180 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.469359:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187596544:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201685.469385:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24c6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c85 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.469405:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24c6 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xe5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c85 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.469413:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24c6 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xe5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c85 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.469423:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24c6 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3c85 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.469433:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24c6 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c85 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.469444:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24c6 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xe5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3c85 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.469452:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24c6 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xe5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3c85 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.469459:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24c6 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3c85 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201685.469686:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d4a80 x1875062187596544/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201685.469698:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24c6 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c85 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788201685.469813:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24c6 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c85 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201685.469826:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/229 (0 0) 00000080:00200000:0.0:1788201685.469832:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788201685.469870:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xe5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788201685.469880:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788201685.469904:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xe5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788201685.469958:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xe5:0x0](ffff8e8289e41988): rc = 0 00000080:00200000:0.0:1788201685.470099:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xe5:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:0.0:1788201685.470106:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xe5:0x0](ffff8e8289e41988) for lock 0x9ae3109cee9e24c6 00000080:00010000:0.0:1788201685.470117:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xe5:0x0](ffff8e8289e41988) 00000080:00010000:0.0:1788201685.470121:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24c6 from it ffffaa1801973e00 00010000:00010000:0.0:1788201685.470124:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24c6 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3c85 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201685.470133:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24c6 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3c85 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201685.470164:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:0.0:1788201685.470180:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788201685.470190:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e24bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c69 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201685.470200:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e24bf lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c69 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201685.470212:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788201685.470216:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201685.470219:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e24bf 00000080:00010000:0.0:1788201685.470223:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24bf from it ffffaa1801973d70 00010000:00010000:0.0:1788201685.470225:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e24bf lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3c69 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201685.470233:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e24bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3c69 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788201685.470241:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:0.0:1788201685.470248:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-13, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:0.0:1788201685.470272:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e24bf lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3c69 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201685.470294:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24c6 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3c85 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201685.470304:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xe5:0x0]: losing layout lock 00010000:00010000:0.0:1788201685.470355:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24c6 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xe5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3c85 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201685.470363:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e24bf lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3c69 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788201685.470368:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788201685.470371:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462000/0x9ae3109cee9e24c6 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xe5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3c85 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788201685.470388:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e24bf lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3c69 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:0.0:1788201685.470407:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788201685.470414:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828793ea00 x1875062187596800/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201685.470436:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ea00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187596800:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:0.0:1788201685.470844:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.474693:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ea00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187596800:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201685.474750:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xe5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201685.474761:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201685.474788:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xe5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201685.474795:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xe5:0x0](ffff8e8289e41988) 00000080:00200000:1.0:1788201685.474886:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201685.474907:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201685.474977:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24cd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.474987:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24cd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.475148:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24cd lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201685.475170:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722a00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187596928:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201685.475243:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64a840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.478887:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722a00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187596928:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201685.478913:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24cd lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3caf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.479119:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24cd lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3caf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.479130:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24cd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3caf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.479141:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24cd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3caf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.479155:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24cd lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3caf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.479162:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24cd lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3caf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.479169:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24cd lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3caf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201685.479180:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d722a00 x1875062187596928/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201685.479198:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24cd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3caf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.479211:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201685.479217:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201685.479243:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201685.479248:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e24cd 00000080:00010000:1.0:1788201685.479253:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201685.479256:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24cd from it ffffaa1801973c10 00010000:00010000:1.0:1788201685.479260:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24cd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3caf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.479267:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24cd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3caf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.479289:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201685.479343:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-14, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201685.479348:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-14, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201685.479367:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-14,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201685.479403:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e24d4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.479410:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e24d4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.479418:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e24d4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201685.479439:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722a00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187597056:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201685.479769:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64a840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.482956:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722a00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187597056:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201685.482993:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e24d4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ccb expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.483020:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e24d4 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xe6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ccb expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.483028:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e24d4 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xe6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ccb expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.483040:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e24d4 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3ccb expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.483050:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e24d4 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ccb expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.483062:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e24d4 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xe6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3ccb expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.483071:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e24d4 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xe6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3ccb expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.483078:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e24d4 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3ccb expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201685.483096:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d722a00 x1875062187597056/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201685.483107:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e24d4 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ccb expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201685.483119:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e24d4 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ccb expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.483128:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/230 (0 0) 00000080:00200000:1.0:1788201685.483135:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201685.483175:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xe6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201685.483185:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201685.483209:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xe6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201685.483262:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xe6:0x0](ffff8e8289e45b88): rc = 0 00000080:00200000:1.0:1788201685.483276:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xe6:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:1.0:1788201685.483281:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xe6:0x0](ffff8e8289e45b88) for lock 0x9ae3109cee9e24d4 00000080:00010000:1.0:1788201685.483291:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xe6:0x0](ffff8e8289e45b88) 00000080:00010000:1.0:1788201685.483295:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24d4 from it ffffaa1801973e00 00010000:00010000:1.0:1788201685.483299:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e24d4 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3ccb expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.483337:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e24d4 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3ccb expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.483366:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201685.483381:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201685.483390:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24cd lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3caf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.483401:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24cd lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3caf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.483407:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201685.483411:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201685.483413:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e24cd 00000080:00010000:1.0:1788201685.483417:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24cd from it ffffaa1801973d70 00010000:00010000:1.0:1788201685.483420:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24cd lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3caf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.483427:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24cd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3caf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201685.483434:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201685.483441:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-14, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201685.483462:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24cd lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3caf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201685.483490:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e24d4 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3ccb expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.483499:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xe6:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.483524:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e24d4 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xe6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3ccb expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.483532:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24cd lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3caf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201685.483546:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201685.483550:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993c00/0x9ae3109cee9e24d4 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xe6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3ccb expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.483564:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24cd lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3caf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201685.483581:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201685.483588:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d722a00 x1875062187597312/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201685.483616:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722a00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187597312:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201685.483694:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64a480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.488959:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722a00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187597312:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201685.489176:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xe6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201685.489188:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201685.489213:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xe6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201685.489220:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xe6:0x0](ffff8e8289e45b88) 00000080:00200000:1.0:1788201685.489333:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201685.489353:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201685.489411:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24db lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.489421:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24db lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.489430:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24db lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201685.489449:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187597440:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201685.489764:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64a840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.499846:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187597440:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201685.499873:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24db lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3cf5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.499893:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24db lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3cf5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.499904:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24db lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3cf5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.499914:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24db lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3cf5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.499927:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24db lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3cf5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.499935:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24db lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3cf5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.499941:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24db lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3cf5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201685.499951:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d720700 x1875062187597440/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201685.500128:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24db lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3cf5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.500141:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201685.500147:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201685.500173:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201685.500179:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e24db 00000080:00010000:1.0:1788201685.500184:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201685.500187:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24db from it ffffaa1801973c10 00010000:00010000:1.0:1788201685.500190:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24db lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3cf5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.500198:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24db lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3cf5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.500221:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201685.500245:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-15, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201685.500250:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-15, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201685.500270:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-15,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201685.500330:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e24e2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.500338:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e24e2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.500346:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e24e2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201685.500370:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187597568:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201685.500557:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64a840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.503936:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187597568:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201685.503960:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e24e2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d11 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.503990:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e24e2 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xe7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d11 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.504004:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e24e2 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xe7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d11 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.504014:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e24e2 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d11 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.504023:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e24e2 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d11 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.504035:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e24e2 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xe7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d11 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.504042:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e24e2 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xe7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d11 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.504050:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e24e2 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d11 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201685.504066:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d720700 x1875062187597568/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201685.504076:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e24e2 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d11 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201685.504087:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e24e2 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d11 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.504096:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/231 (0 0) 00000080:00200000:1.0:1788201685.504102:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201685.504142:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xe7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201685.504154:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201685.504180:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xe7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201685.504235:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xe7:0x0](ffff8e8289e463c8): rc = 0 00000080:00200000:1.0:1788201685.504250:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xe7:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:1.0:1788201685.504255:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xe7:0x0](ffff8e8289e463c8) for lock 0x9ae3109cee9e24e2 00000080:00010000:1.0:1788201685.504266:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xe7:0x0](ffff8e8289e463c8) 00000080:00010000:1.0:1788201685.504269:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24e2 from it ffffaa1801973e00 00010000:00010000:1.0:1788201685.504272:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e24e2 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3d11 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.504280:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e24e2 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3d11 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.504343:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201685.504359:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201685.504368:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3cf5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.504378:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24db lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3cf5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.504384:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201685.504388:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201685.504391:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e24db 00000080:00010000:1.0:1788201685.504395:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24db from it ffffaa1801973d70 00010000:00010000:1.0:1788201685.504398:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24db lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3cf5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.504404:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24db lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3cf5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201685.504412:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201685.504419:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-15, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201685.504442:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24db lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3cf5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201685.504472:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e24e2 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3d11 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.504482:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xe7:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.504509:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e24e2 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xe7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3d11 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.504516:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24db lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3cf5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201685.504522:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201685.504525:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e24e2 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xe7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3d11 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.504548:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24db lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3cf5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201685.504564:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201685.504570:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d720700 x1875062187597824/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201685.504589:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187597824:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201685.504815:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64a480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.508524:0:2401:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788201685.508559:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187597952:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201685.511046:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187597952:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201685.511863:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187597824:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201685.511918:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xe7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201685.511932:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201685.511957:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xe7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201685.511977:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xe7:0x0](ffff8e8289e463c8) 00000080:00200000:1.0:1788201685.512081:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201685.512100:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201685.512154:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24e9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.512164:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24e9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.512175:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24e9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201685.512195:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187598080:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201685.512258:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64a840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.516853:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187598080:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201685.516882:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24e9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d3b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.516894:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24e9 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d3b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.516906:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24e9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d3b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.516917:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24e9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d3b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.516929:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24e9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d3b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.516938:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24e9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d3b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.516945:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24e9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d3b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201685.516973:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d723480 x1875062187598080/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201685.516990:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24e9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d3b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.517015:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201685.517024:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201685.517060:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201685.517068:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e24e9 00000080:00010000:1.0:1788201685.517075:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201685.517080:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24e9 from it ffffaa1801973c10 00010000:00010000:1.0:1788201685.517084:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24e9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d3b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.517094:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24e9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d3b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.517127:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201685.517159:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-16, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201685.517165:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-16, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201685.517198:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-16,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201685.517247:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24f0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.517255:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24f0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.517270:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24f0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201685.517297:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187598208:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201685.517426:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64a840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.521892:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187598208:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201685.521927:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24f0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d57 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.521952:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24f0 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xe8:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d57 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.521972:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24f0 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xe8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d57 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.521984:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24f0 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d57 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.522001:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24f0 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d57 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.522014:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24f0 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xe8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d57 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.522021:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24f0 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xe8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d57 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.522034:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24f0 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d57 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201685.522045:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d723480 x1875062187598208/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201685.522056:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24f0 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d57 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201685.522069:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24f0 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d57 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.522079:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/232 (0 0) 00000080:00200000:1.0:1788201685.522085:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201685.522129:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xe8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201685.522140:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201685.522168:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xe8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201685.522231:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xe8:0x0](ffff8e8289e5ba88): rc = 0 00000080:00200000:1.0:1788201685.522246:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xe8:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:1.0:1788201685.522251:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xe8:0x0](ffff8e8289e5ba88) for lock 0x9ae3109cee9e24f0 00000080:00010000:1.0:1788201685.522261:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xe8:0x0](ffff8e8289e5ba88) 00000080:00010000:1.0:1788201685.522264:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24f0 from it ffffaa1801973e00 00010000:00010000:1.0:1788201685.522267:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24f0 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3d57 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.522276:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24f0 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3d57 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.522377:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201685.522400:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201685.522412:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d3b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.522425:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24e9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d3b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.522430:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201685.522435:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201685.522438:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e24e9 00000080:00010000:1.0:1788201685.522442:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24e9 from it ffffaa1801973d70 00010000:00010000:1.0:1788201685.522445:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24e9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d3b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.522452:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24e9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d3b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201685.522460:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201685.522480:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-16, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201685.522509:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24e9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3d3b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201685.522532:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24f0 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3d57 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.522541:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xe8:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.522576:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24f0 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xe8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3d57 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.522584:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24e9 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3d3b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201685.522591:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201685.522593:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e24f0 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xe8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3d57 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.522609:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24e9 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3d3b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201685.523018:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201685.523030:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d723480 x1875062187598464/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201685.523057:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187598464:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201685.523138:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64a480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.531533:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187598464:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201685.531795:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xe8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201685.531810:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201685.531838:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xe8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201685.531845:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xe8:0x0](ffff8e8289e5ba88) 00000080:00200000:1.0:1788201685.531940:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201685.531961:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201685.532163:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24f7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.532178:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24f7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.532188:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24f7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201685.532208:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187598592:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201685.532387:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64a840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.536872:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187598592:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201685.536900:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24f7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d81 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.536912:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24f7 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d81 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.536926:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24f7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d81 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.536937:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24f7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d81 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.536949:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24f7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d81 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.536956:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24f7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d81 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.537080:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24f7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d81 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201685.537092:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d723100 x1875062187598592/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201685.537107:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24f7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d81 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.537121:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201685.537130:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201685.537163:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201685.537170:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e24f7 00000080:00010000:1.0:1788201685.537176:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201685.537179:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24f7 from it ffffaa1801973c10 00010000:00010000:1.0:1788201685.537182:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24f7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d81 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.537191:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24f7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d81 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.537217:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201685.537244:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-17, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201685.537249:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-17, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201685.537275:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-17,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201685.537420:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24fe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.537428:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24fe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.537438:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24fe lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201685.537460:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187598720:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201685.538197:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64a840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.545841:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187598720:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201685.545869:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24fe lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d9d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.545892:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24fe lrc: 4/1,0 mode: --/PR res: [0x200006991:0xe9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d9d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.545901:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24fe lrc: 4/1,0 mode: --/PR res: [0x200006991:0xe9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d9d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.545911:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24fe lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d9d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.545922:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24fe lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d9d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.545934:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24fe lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xe9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d9d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.545946:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24fe lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xe9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d9d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.545952:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24fe lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xe9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d9d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201685.546241:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d723100 x1875062187598720/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201685.546257:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24fe lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d9d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201685.546271:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24fe lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d9d expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.546282:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/233 (0 0) 00000080:00200000:1.0:1788201685.546288:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201685.546360:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xe9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201685.546372:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201685.546402:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xe9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201685.546460:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xe9:0x0](ffff8e8289e421c8): rc = 0 00000080:00200000:1.0:1788201685.546731:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xe9:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:1.0:1788201685.546740:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xe9:0x0](ffff8e8289e421c8) for lock 0x9ae3109cee9e24fe 00000080:00010000:1.0:1788201685.546752:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xe9:0x0](ffff8e8289e421c8) 00000080:00010000:1.0:1788201685.546756:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24fe from it ffffaa1801973e00 00010000:00010000:1.0:1788201685.546760:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24fe lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xe9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3d9d expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.546775:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24fe lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3d9d expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.546807:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201685.546826:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201685.546836:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24f7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d81 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.546847:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24f7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d81 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.546853:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201685.546858:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201685.546861:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e24f7 00000080:00010000:1.0:1788201685.546864:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e24f7 from it ffffaa1801973d70 00010000:00010000:1.0:1788201685.546867:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24f7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3d81 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.546874:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24f7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3d81 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201685.546881:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201685.546888:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-17, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201685.546913:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24f7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3d81 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201685.546934:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24fe lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xe9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3d9d expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.546943:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xe9:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.547188:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24fe lrc: 1/0,0 mode: --/PR res: [0x200006991:0xe9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3d9d expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.547196:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24f7 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3d81 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201685.547202:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201685.547205:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993e00/0x9ae3109cee9e24fe lrc: 0/0,0 mode: --/PR res: [0x200006991:0xe9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3d9d expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.547222:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e24f7 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3d81 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201685.547242:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201685.547251:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d723100 x1875062187598976/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201685.547276:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187598976:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201685.548336:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64a480 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.557109:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187598976:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201685.557185:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xe9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201685.557200:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201685.557233:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xe9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201685.557240:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xe9:0x0](ffff8e8289e421c8) 00000080:00200000:1.0:1788201685.557388:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201685.557412:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201685.557486:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e2505 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.557497:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e2505 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.557507:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e2505 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201685.557531:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187599104:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201685.557752:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64a840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.560902:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187599104:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201685.560930:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e2505 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3dc7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.560943:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e2505 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3dc7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.560955:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e2505 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3dc7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.560975:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e2505 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3dc7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.560989:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e2505 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3dc7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.561007:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e2505 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3dc7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.561015:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e2505 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3dc7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201685.561028:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d723b80 x1875062187599104/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201685.561041:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e2505 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3dc7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.561052:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201685.561058:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201685.561086:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201685.561092:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2505 00000080:00010000:1.0:1788201685.561098:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201685.561102:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2505 from it ffffaa1801973c10 00010000:00010000:1.0:1788201685.561105:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e2505 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3dc7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.561114:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e2505 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3dc7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.561140:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201685.561162:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-18, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201685.561166:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-18, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201685.561189:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-18,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201685.561230:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e250c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.561237:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e250c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.561246:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e250c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201685.561267:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187599232:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201685.561368:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64a840 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.566902:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187599232:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201685.566931:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e250c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3de3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.566954:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e250c lrc: 4/1,0 mode: --/PR res: [0x200006991:0xea:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3de3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.566973:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e250c lrc: 4/1,0 mode: --/PR res: [0x200006991:0xea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3de3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.566985:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e250c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3de3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.567130:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e250c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3de3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.567144:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e250c lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3de3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.567157:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e250c lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3de3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.567164:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e250c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3de3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201685.567175:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d723b80 x1875062187599232/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201685.567186:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e250c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3de3 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201685.567199:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e250c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3de3 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.567210:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/234 (0 0) 00000080:00200000:1.0:1788201685.567216:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201685.567259:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xea:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201685.567272:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201685.567302:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xea:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201685.567393:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xea:0x0](ffff8e8289e43a88): rc = 0 00000080:00200000:1.0:1788201685.567406:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xea:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:1.0:1788201685.567412:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xea:0x0](ffff8e8289e43a88) for lock 0x9ae3109cee9e250c 00000080:00010000:1.0:1788201685.567422:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xea:0x0](ffff8e8289e43a88) 00000080:00010000:1.0:1788201685.567425:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e250c from it ffffaa1801973e00 00010000:00010000:1.0:1788201685.567428:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e250c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3de3 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.567443:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e250c lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3de3 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.567483:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201685.567498:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201685.567506:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e2505 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3dc7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.567516:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e2505 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3dc7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.567522:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201685.567526:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201685.567528:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2505 00000080:00010000:1.0:1788201685.567532:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2505 from it ffffaa1801973d70 00010000:00010000:1.0:1788201685.567534:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e2505 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3dc7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.567740:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e2505 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3dc7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201685.567749:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201685.567756:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-18, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201685.567784:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e2505 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3dc7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201685.567806:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e250c lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xea:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3de3 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.567815:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xea:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.567842:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e250c lrc: 1/0,0 mode: --/PR res: [0x200006991:0xea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3de3 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.567849:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e2505 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3dc7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201685.567854:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201685.567857:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e250c lrc: 0/0,0 mode: --/PR res: [0x200006991:0xea:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3de3 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.567873:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e2505 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3dc7 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201685.567890:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201685.567898:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d723b80 x1875062187599488/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201685.567919:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187599488:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201685.568449:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64a480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.579577:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187599488:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.579907:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xea:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.579922:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.579951:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xea:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.579959:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xea:0x0](ffff8e8289e43a88) 00000080:00200000:2.0:1788201685.580372:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.580401:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.580822:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2513 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.580833:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2513 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.580843:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2513 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.580869:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187599616:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.580967:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82922303c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.585436:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187599616:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.585529:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2513 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e0d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.585550:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2513 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e0d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.585562:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2513 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e0d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.585573:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2513 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e0d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.585585:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2513 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e0d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.585594:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2513 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e0d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.585601:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2513 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e0d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.585613:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d4a80 x1875062187599616/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.585628:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2513 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e0d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.585648:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.585655:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.585680:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.585687:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2513 00000080:00010000:2.0:1788201685.585691:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.585695:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2513 from it ffffaa1801973c10 00010000:00010000:2.0:1788201685.585698:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2513 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e0d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.585706:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2513 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e0d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.585727:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.585748:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-19, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.585753:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-19, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.585772:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-19,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.585811:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e251a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.585818:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e251a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.585827:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e251a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.585844:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187599744:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.585972:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82922303c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.591365:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187599744:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.591393:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e251a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e29 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.591420:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e251a lrc: 4/1,0 mode: --/PR res: [0x200006991:0xeb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e29 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.591429:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e251a lrc: 4/1,0 mode: --/PR res: [0x200006991:0xeb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e29 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.591441:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e251a lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xeb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e29 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.591450:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e251a lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xeb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e29 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.591461:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e251a lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xeb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e29 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.591475:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e251a lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xeb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e29 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.591479:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e251a lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xeb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e29 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.591488:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d4a80 x1875062187599744/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.591497:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e251a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xeb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e29 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.591510:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e251a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xeb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e29 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.591517:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/235 (0 0) 00000080:00200000:2.0:1788201685.591522:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.591562:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xeb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.591570:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.591593:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xeb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.591655:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xeb:0x0](ffff8e8289e442c8): rc = 0 00000080:00200000:2.0:1788201685.591670:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xeb:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.591676:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xeb:0x0](ffff8e8289e442c8) for lock 0x9ae3109cee9e251a 00000080:00010000:2.0:1788201685.591685:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xeb:0x0](ffff8e8289e442c8) 00000080:00010000:2.0:1788201685.591689:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e251a from it ffffaa1801973e00 00010000:00010000:2.0:1788201685.591692:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e251a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xeb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3e29 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.591701:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e251a lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xeb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3e29 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.591733:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.591749:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.591757:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2513 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e0d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.591766:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2513 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e0d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.591773:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.591777:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.591779:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2513 00000080:00010000:2.0:1788201685.591782:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2513 from it ffffaa1801973d70 00010000:00010000:2.0:1788201685.591784:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2513 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e0d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.591788:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2513 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e0d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.591793:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.591798:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-19, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.591815:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2513 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3e0d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.591831:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e251a lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xeb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3e29 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.591836:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xeb:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.591859:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e251a lrc: 1/0,0 mode: --/PR res: [0x200006991:0xeb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3e29 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.591864:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2513 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3e0d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.591867:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.591869:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e251a lrc: 0/0,0 mode: --/PR res: [0x200006991:0xeb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3e29 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.591880:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2513 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3e0d expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.591891:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.591896:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894d4a80 x1875062187600000/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.591914:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187600000:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201685.594606:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230600 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.601158:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187600000:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201685.601218:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xeb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201685.601231:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201685.601257:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xeb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201685.601265:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xeb:0x0](ffff8e8289e442c8) 00000080:00200000:3.0:1788201685.601370:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201685.601388:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201685.601437:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2521 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.601444:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2521 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.601451:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2521 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201685.601465:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187600128:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201685.601532:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1d80 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.604822:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187600128:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201685.604846:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2521 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e53 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.604858:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2521 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e53 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.604873:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2521 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e53 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.604883:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2521 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e53 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.604893:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2521 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e53 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.604901:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2521 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e53 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.604908:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2521 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e53 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201685.604917:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289539880 x1875062187600128/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201685.604929:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2521 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e53 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.604940:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201685.604946:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201685.604970:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201685.604976:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2521 00000080:00010000:3.0:1788201685.604980:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201685.604984:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2521 from it ffffaa1801973c10 00010000:00010000:3.0:1788201685.604987:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2521 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e53 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.604996:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2521 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e53 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.605018:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201685.605051:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-20, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201685.605056:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-20, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201685.605075:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-20,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201685.605125:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e2528 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.605133:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e2528 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.605185:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e2528 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201685.605205:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187600256:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201685.605269:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1d80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.618954:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187600256:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201685.619297:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e2528 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e6f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.619337:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e2528 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xec:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e6f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.619351:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e2528 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e6f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.619361:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e2528 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e6f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.619371:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e2528 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e6f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.619384:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e2528 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e6f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.619392:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e2528 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e6f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.619399:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e2528 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e6f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201685.619410:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289539880 x1875062187600256/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201685.619420:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e2528 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e6f expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201685.619432:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e2528 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e6f expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.619448:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/236 (0 0) 00000080:00200000:1.0:1788201685.619455:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201685.619501:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xec:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201685.619513:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201685.619550:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xec:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201685.619618:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xec:0x0](ffff8e8289c0db88): rc = 0 00000080:00200000:1.0:1788201685.619633:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xec:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:1.0:1788201685.619652:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xec:0x0](ffff8e8289c0db88) for lock 0x9ae3109cee9e2528 00000080:00010000:1.0:1788201685.619663:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xec:0x0](ffff8e8289c0db88) 00000080:00010000:1.0:1788201685.619666:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2528 from it ffffaa1801973e00 00010000:00010000:1.0:1788201685.619670:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e2528 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3e6f expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.619679:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e2528 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3e6f expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.619711:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201685.619728:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201685.619737:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2521 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e53 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.619747:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2521 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e53 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.619755:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201685.619759:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201685.619762:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2521 00000080:00010000:1.0:1788201685.619766:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2521 from it ffffaa1801973d70 00010000:00010000:1.0:1788201685.619769:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2521 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e53 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.619776:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2521 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e53 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201685.619784:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201685.619790:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-20, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201685.619814:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2521 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3e53 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201685.619836:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e2528 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3e6f expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.619846:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xec:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.619874:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e2528 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3e6f expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.619880:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2521 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3e53 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201685.619885:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201685.619888:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e2528 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xec:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3e6f expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.619903:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2521 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3e53 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201685.619925:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201685.619931:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d721f80 x1875062187600512/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201685.619952:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187600512:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201685.620037:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64aa80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.624992:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187600512:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201685.625061:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xec:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201685.625075:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201685.625102:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xec:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201685.625110:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xec:0x0](ffff8e8289c0db88) 00000080:00200000:1.0:1788201685.625200:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201685.625220:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201685.625273:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e252f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.625283:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e252f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.625292:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e252f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201685.625355:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187600640:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201685.625414:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64a9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.628724:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187600640:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201685.628754:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e252f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e99 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.628768:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e252f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e99 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.628779:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e252f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e99 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.628791:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e252f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e99 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.628803:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e252f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e99 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.628812:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e252f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e99 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.628818:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e252f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e99 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201685.628830:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d721880 x1875062187600640/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201685.628845:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e252f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e99 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.628856:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201685.628863:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788201685.628890:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:1.0:1788201685.628897:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e252f 00000080:00010000:1.0:1788201685.628902:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201685.628906:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e252f from it ffffaa1801973c10 00010000:00010000:1.0:1788201685.628909:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e252f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e99 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.628917:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e252f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e99 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.628943:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:1.0:1788201685.629436:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-21, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:1.0:1788201685.629443:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-21, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:1.0:1788201685.629468:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-21,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788201685.629517:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e2536 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.629527:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e2536 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.629605:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e2536 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201685.629707:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187600768:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201685.629816:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64a9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.632587:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187600768:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201685.632624:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e2536 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3eb5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.632659:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e2536 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xed:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3eb5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.632667:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e2536 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3eb5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.632678:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e2536 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3eb5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.632688:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e2536 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3eb5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.632701:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e2536 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3eb5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.632709:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e2536 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3eb5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.632717:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e2536 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3eb5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201685.632730:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d721880 x1875062187600768/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201685.632740:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e2536 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3eb5 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788201685.632758:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e2536 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3eb5 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.632768:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/237 (0 0) 00000080:00200000:1.0:1788201685.632775:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788201685.632816:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xed:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201685.632827:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201685.632856:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xed:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201685.632980:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xed:0x0](ffff8e8289e5a1c8): rc = 0 00000080:00200000:1.0:1788201685.633011:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xed:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:1.0:1788201685.633017:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xed:0x0](ffff8e8289e5a1c8) for lock 0x9ae3109cee9e2536 00000080:00010000:1.0:1788201685.633027:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xed:0x0](ffff8e8289e5a1c8) 00000080:00010000:1.0:1788201685.633030:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2536 from it ffffaa1801973e00 00010000:00010000:1.0:1788201685.633035:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e2536 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3eb5 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.633043:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e2536 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3eb5 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.633071:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201685.633087:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201685.633097:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e252f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e99 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.633107:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e252f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e99 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.633119:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788201685.633123:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:1.0:1788201685.633126:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e252f 00000080:00010000:1.0:1788201685.633131:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e252f from it ffffaa1801973d70 00010000:00010000:1.0:1788201685.633134:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e252f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3e99 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.633141:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e252f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3e99 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788201685.633149:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:1.0:1788201685.633156:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-21, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:1.0:1788201685.633180:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e252f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3e99 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201685.633201:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e2536 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3eb5 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788201685.633210:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xed:0x0]: losing layout lock 00010000:00010000:1.0:1788201685.633239:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e2536 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3eb5 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.633247:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e252f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3e99 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201685.633253:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788201685.633256:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993400/0x9ae3109cee9e2536 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xed:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3eb5 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788201685.633271:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287992600/0x9ae3109cee9e252f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3e99 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788201685.633289:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788201685.633295:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828d721880 x1875062187601024/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201685.633355:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187601024:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:1.0:1788201685.633423:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64aa80 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.642148:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187601024:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:1.0:1788201685.642212:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xed:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788201685.642227:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788201685.642256:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xed:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788201685.642265:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xed:0x0](ffff8e8289e5a1c8) 00000080:00200000:1.0:1788201685.642438:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:1.0:1788201685.642456:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788201685.642511:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e253d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.642522:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e253d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.642532:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e253d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788201685.642694:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187601152:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:1.0:1788201685.642803:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e828d64a9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788201685.646267:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187601152:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:1.0:1788201685.646294:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e253d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3edf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.646394:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e253d lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3edf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.646406:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e253d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3edf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.646418:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e253d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3edf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.646431:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e253d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3edf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.646439:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e253d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3edf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.646447:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e253d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3edf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788201685.646459:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828d723800 x1875062187601152/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788201685.646536:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e253d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3edf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788201685.646565:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788201685.646573:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.646795:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.646804:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e253d 00000080:00010000:2.0:1788201685.646810:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.646813:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e253d from it ffffaa1801973c10 00010000:00010000:2.0:1788201685.646817:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e253d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3edf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.646826:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e253d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3edf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.646850:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.646871:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-22, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.646875:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-22, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.646897:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-22,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.646941:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2544 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.646949:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2544 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.646958:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2544 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.647143:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5c00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187601280:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.647275:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.652648:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5c00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187601280:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.652676:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2544 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3efb expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.652697:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2544 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xee:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3efb expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.652705:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2544 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3efb expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.652716:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2544 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3efb expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.652726:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2544 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3efb expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.652737:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2544 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3efb expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.652746:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2544 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3efb expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.652758:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2544 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3efb expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.652770:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d5c00 x1875062187601280/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.652781:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2544 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3efb expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.652793:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2544 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3efb expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.652803:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/238 (0 0) 00000080:00200000:2.0:1788201685.652809:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.652847:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xee:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.652857:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.652882:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xee:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.652932:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xee:0x0](ffff8e8289e5e3c8): rc = 0 00000080:00200000:2.0:1788201685.652946:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xee:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.652951:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xee:0x0](ffff8e8289e5e3c8) for lock 0x9ae3109cee9e2544 00000080:00010000:2.0:1788201685.652960:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xee:0x0](ffff8e8289e5e3c8) 00000080:00010000:2.0:1788201685.652974:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2544 from it ffffaa1801973e00 00010000:00010000:2.0:1788201685.652978:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2544 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3efb expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.652987:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2544 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3efb expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.653184:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.653201:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.653211:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e253d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3edf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.653221:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e253d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3edf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.653228:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.653232:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.653236:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e253d 00000080:00010000:2.0:1788201685.653240:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e253d from it ffffaa1801973d70 00010000:00010000:2.0:1788201685.653244:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e253d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3edf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.653251:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e253d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3edf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.653259:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.653267:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-22, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.653291:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e253d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3edf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.653350:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2544 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3efb expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.653362:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xee:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.653390:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2544 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3efb expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.653398:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e253d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3edf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.653403:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.653406:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2544 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xee:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3efb expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.653420:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8287993200/0x9ae3109cee9e253d lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3edf expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.653438:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.653445:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894d5c00 x1875062187601536/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.653467:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5c00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187601536:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.653588:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.658161:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5c00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187601536:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.658218:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xee:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.658231:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.658257:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xee:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.658264:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xee:0x0](ffff8e8289e5e3c8) 00000080:00200000:2.0:1788201685.658387:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.658407:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.658460:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e254b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.658475:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e254b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.658485:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e254b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.658503:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187601664:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.658565:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.662518:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187601664:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.662601:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e254b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f25 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.662686:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e254b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f25 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.662698:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e254b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3f25 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.662709:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e254b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f25 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.662722:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e254b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3f25 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.662729:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e254b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3f25 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.662742:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e254b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3f25 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.662753:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d4e00 x1875062187601664/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.662767:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e254b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f25 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.662780:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.662787:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.662817:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.662824:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e254b 00000080:00010000:2.0:1788201685.662830:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.662833:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e254b from it ffffaa1801973c10 00010000:00010000:2.0:1788201685.662836:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e254b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3f25 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.662846:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e254b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f25 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.662868:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.662890:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-23, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.662894:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-23, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.662913:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-23,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.662950:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2552 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.662958:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2552 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.662981:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2552 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.663010:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187601792:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.663081:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.666712:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187601792:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.666739:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2552 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f41 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.666759:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2552 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xef:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f41 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.666768:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2552 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f41 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.666779:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2552 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3f41 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.666789:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2552 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f41 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.666800:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2552 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3f41 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.666813:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2552 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3f41 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.666820:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2552 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3f41 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.666830:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d4e00 x1875062187601792/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.666841:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2552 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f41 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.666852:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2552 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f41 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.666861:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/239 (0 0) 00000080:00200000:2.0:1788201685.666867:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.666905:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xef:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.666915:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.666938:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xef:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.667126:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xef:0x0](ffff8e8289e5f448): rc = 0 00000080:00200000:2.0:1788201685.667143:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xef:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.667149:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xef:0x0](ffff8e8289e5f448) for lock 0x9ae3109cee9e2552 00000080:00010000:2.0:1788201685.667159:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xef:0x0](ffff8e8289e5f448) 00000080:00010000:2.0:1788201685.667162:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2552 from it ffffaa1801973e00 00010000:00010000:2.0:1788201685.667165:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2552 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3f41 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.667178:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2552 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3f41 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.667207:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.667222:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.667232:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e254b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f25 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.667242:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e254b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f25 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.667248:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.667252:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.667254:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e254b 00000080:00010000:2.0:1788201685.667258:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e254b from it ffffaa1801973d70 00010000:00010000:2.0:1788201685.667261:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e254b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3f25 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.667267:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e254b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f25 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.667274:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.667280:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-23, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.667303:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e254b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3f25 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.667370:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2552 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3f41 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.667379:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xef:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.667408:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2552 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3f41 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.667415:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e254b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3f25 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.667420:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.667422:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2552 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xef:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3f41 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.667436:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e254b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3f25 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.667451:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.667458:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894d4e00 x1875062187602048/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.667479:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187602048:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.667732:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.672181:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187602048:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.672242:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xef:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.672255:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.672283:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xef:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.672290:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xef:0x0](ffff8e8289e5f448) 00000080:00200000:2.0:1788201685.672469:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.672489:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.672714:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2559 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.672726:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2559 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.672737:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2559 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.672761:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6d80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187602176:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.672884:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.675897:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6d80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187602176:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.675923:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2559 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f6b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.675936:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2559 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f6b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.675947:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2559 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3f6b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.675957:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2559 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f6b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.676125:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2559 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3f6b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.676133:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2559 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3f6b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.676142:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2559 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3f6b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.676155:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d6d80 x1875062187602176/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.676168:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2559 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f6b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.676181:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.676188:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.676217:0:42638:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.676225:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2559 00000080:00010000:2.0:1788201685.676231:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.676235:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2559 from it ffffaa1801973c10 00010000:00010000:2.0:1788201685.676239:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2559 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3f6b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.676249:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2559 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f6b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.676276:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.676303:0:42638:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-24, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.676393:0:42638:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-24, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.676416:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-24,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.676458:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2560 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.676466:0:42638:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2560 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.676476:0:42638:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2560 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42638 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.676495:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6d80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187602304:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.676621:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.679959:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6d80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187602304:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.679983:0:42638:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2560 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f87 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.680049:0:42638:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2560 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xf0:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f87 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.680058:0:42638:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2560 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xf0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f87 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.680070:0:42638:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2560 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3f87 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.680080:0:42638:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2560 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f87 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.680093:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2560 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xf0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3f87 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.680105:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2560 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xf0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3f87 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.680113:0:42638:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2560 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3f87 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.680124:0:42638:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d6d80 x1875062187602304/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.680135:0:42638:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2560 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f87 expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.680148:0:42638:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2560 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f87 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.680158:0:42638:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/240 (0 0) 00000080:00200000:2.0:1788201685.680164:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.680205:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xf0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.680216:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.680241:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xf0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.680293:0:42638:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xf0:0x0](ffff8e8289e45348): rc = 0 00000080:00200000:2.0:1788201685.680376:0:42638:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xf0:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.680383:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xf0:0x0](ffff8e8289e45348) for lock 0x9ae3109cee9e2560 00000080:00010000:2.0:1788201685.680392:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xf0:0x0](ffff8e8289e45348) 00000080:00010000:2.0:1788201685.680396:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2560 from it ffffaa1801973e00 00010000:00010000:2.0:1788201685.680399:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2560 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3f87 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.680413:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2560 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3f87 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.680439:0:42638:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.680454:0:42638:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.680463:0:42638:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2559 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f6b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.680473:0:42638:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2559 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f6b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.680480:0:42638:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.680484:0:42638:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.680487:0:42638:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2559 00000080:00010000:2.0:1788201685.680490:0:42638:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2559 from it ffffaa1801973d70 00010000:00010000:2.0:1788201685.680493:0:42638:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2559 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3f6b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.680500:0:42638:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2559 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3f6b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.680508:0:42638:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.680515:0:42638:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-24, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.680681:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2559 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3f6b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.680706:0:42638:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2560 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf0:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3f87 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.680716:0:42638:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xf0:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.680744:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2560 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xf0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3f87 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.680751:0:42638:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2559 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3f6b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.680756:0:42638:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.680759:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2560 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xf0:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3f87 expref: -99 pid: 42638 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.680773:0:42638:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2559 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3f6b expref: -99 pid: 42638 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.680788:0:42638:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.680795:0:42638:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894d6d80 x1875062187602560/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.680816:0:42638:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d6d80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187602560:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.680923:0:42638:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.685511:0:42638:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d6d80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42638:1875062187602560:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.685577:0:42638:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xf0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.685589:0:42638:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.685623:0:42638:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xf0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.685630:0:42638:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xf0:0x0](ffff8e8289e45348) 00000080:00200000:3.0:1788201685.731161:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201685.731191:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201685.731254:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2567 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.731266:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2567 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.731277:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2567 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201685.731305:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187602688:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201685.731783:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.734555:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187602688:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201685.734583:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2567 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3fb1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.734595:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2567 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3fb1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.735714:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2567 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3fb1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.735729:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2567 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3fb1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.735745:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2567 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3fb1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.735754:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2567 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3fb1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.735762:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2567 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3fb1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201685.735777:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828953a680 x1875062187602688/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201685.735796:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3fb1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.735809:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201685.735817:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788201685.736087:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:0.0:1788201685.736095:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2567 00000080:00010000:0.0:1788201685.736101:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:0.0:1788201685.736104:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2567 from it ffffaa180174bc10 00010000:00010000:0.0:1788201685.736108:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3fb1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201685.736117:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3fb1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788201685.736141:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:0.0:1788201685.736164:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-0, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:0.0:1788201685.736168:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-0, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:0.0:1788201685.736187:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-0,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788201685.736230:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e256e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201685.736238:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e256e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788201685.736247:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e256e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788201685.736268:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187602816:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:0.0:1788201685.736891:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8291165000 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.740272:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187602816:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.740300:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e256e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3fcd expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.740362:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e256e lrc: 4/1,0 mode: --/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3fcd expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.740370:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e256e lrc: 4/1,0 mode: --/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3fcd expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.740381:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e256e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3fcd expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.740390:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e256e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3fcd expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.740402:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e256e lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3fcd expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.740410:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e256e lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3fcd expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.740417:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e256e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3fcd expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.740429:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828793d500 x1875062187602816/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.740440:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e256e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3fcd expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.740454:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e256e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3fcd expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.740924:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/241 (0 0) 00000080:00200000:2.0:1788201685.740934:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.740980:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xf1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.740991:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.741027:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xf1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.741084:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xf1:0x0](ffff8e8289e58908): rc = 0 00000080:00200000:2.0:1788201685.741100:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xf1:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.741106:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xf1:0x0](ffff8e8289e58908) for lock 0x9ae3109cee9e256e 00000080:00010000:2.0:1788201685.741116:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xf1:0x0](ffff8e8289e58908) 00000080:00010000:2.0:1788201685.741119:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e256e from it ffffaa180174be00 00010000:00010000:2.0:1788201685.741123:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e256e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc3fcd expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.741132:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e256e lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc3fcd expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.741164:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.741183:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.741193:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3fb1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.741203:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2567 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3fb1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.741215:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.741220:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.741223:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2567 00000080:00010000:2.0:1788201685.741226:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2567 from it ffffaa180174bd70 00010000:00010000:2.0:1788201685.741229:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2567 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3fb1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.741237:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3fb1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.741246:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.741253:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-0, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.741348:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2567 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3fb1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.741372:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e256e lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc3fcd expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.741381:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xf1:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.741412:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e256e lrc: 1/0,0 mode: --/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3fcd expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.741420:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2567 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3fb1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.741427:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.741430:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8290ee1600/0x9ae3109cee9e256e lrc: 0/0,0 mode: --/PR res: [0x200006991:0xf1:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc3fcd expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.741446:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2567 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3fb1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.741475:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.741484:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894d7480 x1875062187603072/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.741504:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187603072:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.741894:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.745962:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187603072:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.746289:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xf1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.746304:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.746718:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xf1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.746727:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xf1:0x0](ffff8e8289e58908) 00000080:00200000:2.0:1788201685.747049:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.747074:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.747131:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2575 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.747142:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2575 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.747152:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2575 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.747173:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187603200:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.747333:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.749869:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187603200:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.749896:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2575 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ff7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.749908:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2575 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ff7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.749919:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2575 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3ff7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.749931:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2575 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ff7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.749943:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2575 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3ff7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.749951:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2575 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3ff7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.749958:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2575 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3ff7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.749970:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d5f80 x1875062187603200/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.749982:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2575 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ff7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.750010:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.750018:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.750043:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.750051:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2575 00000080:00010000:2.0:1788201685.750056:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.750059:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2575 from it ffffaa180174bc10 00010000:00010000:2.0:1788201685.750063:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2575 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3ff7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.750071:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2575 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ff7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.750094:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.750114:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-1, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.750119:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-1, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.750137:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-1,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.750173:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e257c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.750180:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e257c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.750189:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e257c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.750208:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187603328:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.750280:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.760016:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187603328:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.760047:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e257c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4013 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.760072:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e257c lrc: 4/1,0 mode: --/PR res: [0x200006991:0xf2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4013 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.760081:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e257c lrc: 4/1,0 mode: --/PR res: [0x200006991:0xf2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4013 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.760093:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e257c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4013 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.760103:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e257c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4013 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.760116:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e257c lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xf2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4013 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.760125:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e257c lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xf2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4013 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.760135:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e257c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4013 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.760146:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d5f80 x1875062187603328/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.760157:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e257c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4013 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.760172:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e257c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4013 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.760179:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/242 (0 0) 00000080:00200000:2.0:1788201685.760185:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.760222:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xf2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.760231:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.760254:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xf2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.760301:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xf2:0x0](ffff8e8289e41148): rc = 0 00000080:00200000:2.0:1788201685.760343:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xf2:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.760348:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xf2:0x0](ffff8e8289e41148) for lock 0x9ae3109cee9e257c 00000080:00010000:2.0:1788201685.760356:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xf2:0x0](ffff8e8289e41148) 00000080:00010000:2.0:1788201685.760359:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e257c from it ffffaa180174be00 00010000:00010000:2.0:1788201685.760361:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e257c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc4013 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.760368:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e257c lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc4013 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.760392:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.760404:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.760410:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2575 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ff7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.760417:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2575 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ff7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.760425:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.760428:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.760430:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2575 00000080:00010000:2.0:1788201685.760432:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2575 from it ffffaa180174bd70 00010000:00010000:2.0:1788201685.760433:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2575 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc3ff7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.760438:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2575 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc3ff7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.760442:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.760447:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-1, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.760473:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2575 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc3ff7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.760488:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e257c lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc4013 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.760493:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xf2:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.760515:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e257c lrc: 1/0,0 mode: --/PR res: [0x200006991:0xf2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc4013 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.760520:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2575 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3ff7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.760523:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.760525:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e257c lrc: 0/0,0 mode: --/PR res: [0x200006991:0xf2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc4013 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.760538:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2575 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc3ff7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.760555:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.760561:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894d5f80 x1875062187603584/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.760583:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187603584:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.760653:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.764527:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187603584:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.764582:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xf2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.764594:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.764619:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xf2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.764626:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xf2:0x0](ffff8e8289e41148) 00000080:00200000:2.0:1788201685.764722:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.764741:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.764791:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2583 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.764798:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2583 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.764805:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2583 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.764819:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187603712:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.764877:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.767342:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187603712:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.767362:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2583 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc403d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.767371:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2583 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc403d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.767379:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2583 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc403d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.767385:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2583 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc403d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.767393:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2583 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc403d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.767398:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2583 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc403d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.767402:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2583 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc403d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.767409:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d5880 x1875062187603712/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.767418:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2583 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc403d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.767430:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.767434:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.767454:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.767458:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2583 00000080:00010000:2.0:1788201685.767470:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.767472:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2583 from it ffffaa180174bc10 00010000:00010000:2.0:1788201685.767474:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2583 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc403d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.767480:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2583 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc403d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.767498:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.767514:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-2, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.767517:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-2, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.767532:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-2,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.767567:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e258a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.767575:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e258a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.767584:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e258a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.767601:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187603840:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.767671:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.772902:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187603840:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.772924:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e258a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4059 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.772941:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e258a lrc: 4/1,0 mode: --/PR res: [0x200006991:0xf3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4059 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.772949:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e258a lrc: 4/1,0 mode: --/PR res: [0x200006991:0xf3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4059 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.772960:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e258a lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4059 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.772971:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e258a lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4059 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.772984:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e258a lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xf3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4059 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.772991:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e258a lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xf3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4059 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.772999:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e258a lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4059 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.773011:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d5880 x1875062187603840/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.773022:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e258a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4059 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.773035:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e258a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4059 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.773045:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/243 (0 0) 00000080:00200000:2.0:1788201685.773052:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.773094:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xf3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.773105:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.773132:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xf3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.773193:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xf3:0x0](ffff8e8289e3cb08): rc = 0 00000080:00200000:2.0:1788201685.773209:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xf3:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.773214:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xf3:0x0](ffff8e8289e3cb08) for lock 0x9ae3109cee9e258a 00000080:00010000:2.0:1788201685.773224:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xf3:0x0](ffff8e8289e3cb08) 00000080:00010000:2.0:1788201685.773228:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e258a from it ffffaa180174be00 00010000:00010000:2.0:1788201685.773231:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e258a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc4059 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.773240:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e258a lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc4059 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.773265:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.773280:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.773288:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2583 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc403d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.773295:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2583 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc403d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.773303:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.773506:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.773509:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2583 00000080:00010000:2.0:1788201685.773512:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2583 from it ffffaa180174bd70 00010000:00010000:2.0:1788201685.773514:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2583 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc403d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.773519:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2583 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc403d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.773526:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.773531:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-2, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.773556:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2583 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc403d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.773578:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e258a lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc4059 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.773588:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xf3:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.773619:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e258a lrc: 1/0,0 mode: --/PR res: [0x200006991:0xf3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc4059 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.773626:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2583 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc403d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.773631:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.773644:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e258a lrc: 0/0,0 mode: --/PR res: [0x200006991:0xf3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc4059 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.773660:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2583 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc403d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.773676:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.773684:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894d5880 x1875062187604096/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.773706:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187604096:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.773783:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.779480:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5880 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187604096:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.779539:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xf3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.779552:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.779578:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xf3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.779585:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xf3:0x0](ffff8e8289e3cb08) 00000080:00200000:2.0:1788201685.779682:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.779701:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.779752:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2591 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.779762:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2591 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.779772:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2591 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.779790:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187604224:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.779853:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.786098:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187604224:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.786125:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2591 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4083 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.786138:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2591 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4083 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.786149:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2591 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4083 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.786160:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2591 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4083 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.786172:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2591 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4083 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.786181:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2591 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4083 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.786188:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2591 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4083 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.786201:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d4700 x1875062187604224/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.786215:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2591 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4083 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.786233:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.786240:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.786266:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.786272:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2591 00000080:00010000:2.0:1788201685.786278:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.786281:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2591 from it ffffaa180174bc10 00010000:00010000:2.0:1788201685.786285:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2591 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4083 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.786293:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2591 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4083 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.786574:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.786601:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-3, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.786605:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-3, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.786626:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-3,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.786676:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2598 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.786683:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2598 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.786692:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2598 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.786712:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187604352:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.786789:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.789738:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187604352:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.789764:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2598 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc409f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.789786:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2598 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xf4:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc409f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.789794:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2598 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xf4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc409f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.789805:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2598 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc409f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.789815:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2598 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc409f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.789827:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2598 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xf4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc409f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.789835:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2598 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xf4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc409f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.789842:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2598 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc409f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.789854:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d4700 x1875062187604352/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.789865:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2598 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc409f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.789878:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2598 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc409f expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.789888:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/244 (0 0) 00000080:00200000:2.0:1788201685.789895:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.798083:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xf4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.798102:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.798132:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xf4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.798208:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xf4:0x0](ffff8e8289e39988): rc = 0 00000080:00200000:2.0:1788201685.798228:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xf4:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.798238:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xf4:0x0](ffff8e8289e39988) for lock 0x9ae3109cee9e2598 00000080:00010000:2.0:1788201685.798250:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xf4:0x0](ffff8e8289e39988) 00000080:00010000:2.0:1788201685.798254:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2598 from it ffffaa180174be00 00010000:00010000:2.0:1788201685.798259:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2598 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc409f expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.798273:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2598 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc409f expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.798341:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.798362:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.798376:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2591 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4083 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.798387:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2591 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4083 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.798400:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.798404:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.798407:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2591 00000080:00010000:2.0:1788201685.798411:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2591 from it ffffaa180174bd70 00010000:00010000:2.0:1788201685.798414:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2591 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4083 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.798420:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2591 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4083 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.798428:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.798435:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-3, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.798745:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2591 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc4083 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.798772:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2598 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc409f expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.798783:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xf4:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.798818:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2598 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xf4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc409f expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.798826:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2591 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc4083 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.798833:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.798835:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2598 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xf4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc409f expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.798853:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2591 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc4083 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.798869:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.798880:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894d4700 x1875062187604608/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.798905:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d4700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187604608:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.799058:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.811326:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d4700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187604608:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.811390:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xf4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.811403:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.811430:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xf4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.811438:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xf4:0x0](ffff8e8289e39988) 00000080:00200000:3.0:1788201685.812170:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201685.812192:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201685.812248:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e259f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.812259:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e259f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.812269:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e259f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201685.812292:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187604736:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201685.812469:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.820170:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187604736:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201685.820199:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e259f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc40c9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.820212:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e259f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc40c9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.820224:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e259f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc40c9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.820239:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e259f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc40c9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.820251:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e259f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc40c9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.820259:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e259f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc40c9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.820266:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e259f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc40c9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201685.820276:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289539180 x1875062187604736/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201685.820290:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e259f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc40c9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.820302:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201685.820335:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201685.820361:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201685.820366:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e259f 00000080:00010000:3.0:1788201685.820371:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201685.820374:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e259f from it ffffaa180174bc10 00010000:00010000:3.0:1788201685.820377:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e259f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc40c9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.820390:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e259f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc40c9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.820414:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201685.820433:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-4, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201685.820437:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-4, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201685.820458:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-4,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201685.820499:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e25a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.820506:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e25a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.820515:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e25a6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201685.820536:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187604864:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201685.820600:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.824373:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187604864:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201685.824400:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e25a6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc40e5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.824418:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e25a6 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xf5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc40e5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.824426:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e25a6 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xf5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc40e5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.824437:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e25a6 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc40e5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.824448:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e25a6 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc40e5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.824470:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e25a6 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xf5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc40e5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.824478:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e25a6 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xf5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc40e5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.824485:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e25a6 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc40e5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201685.824496:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289539180 x1875062187604864/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201685.824506:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e25a6 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc40e5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201685.824519:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e25a6 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc40e5 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.824529:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/245 (0 0) 00000080:00200000:3.0:1788201685.824705:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201685.824746:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xf5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201685.824758:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201685.824786:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xf5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201685.824844:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xf5:0x0](ffff8e8289e3db88): rc = 0 00000080:00200000:3.0:1788201685.824858:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xf5:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201685.824864:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xf5:0x0](ffff8e8289e3db88) for lock 0x9ae3109cee9e25a6 00000080:00010000:3.0:1788201685.824874:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xf5:0x0](ffff8e8289e3db88) 00000080:00010000:3.0:1788201685.824877:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e25a6 from it ffffaa180174be00 00010000:00010000:3.0:1788201685.824880:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e25a6 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc40e5 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.824890:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e25a6 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc40e5 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.824920:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201685.824936:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201685.824945:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e259f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc40c9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.824956:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e259f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc40c9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.824962:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201685.824967:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201685.824970:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e259f 00000080:00010000:3.0:1788201685.824973:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e259f from it ffffaa180174bd70 00010000:00010000:3.0:1788201685.824976:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e259f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc40c9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.824987:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e259f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc40c9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201685.825051:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201685.825059:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-4, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201685.825084:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e259f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc40c9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201685.825106:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e25a6 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc40e5 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.825115:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xf5:0x0]: losing layout lock 00010000:00010000:3.0:1788201685.825143:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e25a6 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xf5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc40e5 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.825151:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e259f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc40c9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201685.825157:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201685.825160:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e25a6 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xf5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc40e5 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.825174:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e259f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc40c9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201685.825192:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201685.825200:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289539180 x1875062187605120/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201685.825221:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187605120:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201685.825303:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1000 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.828441:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187605120:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201685.828826:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xf5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201685.828839:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201685.828865:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xf5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201685.828873:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xf5:0x0](ffff8e8289e3db88) 00000080:00200000:3.0:1788201685.829125:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201685.829147:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201685.829203:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e25ad lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.829212:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e25ad lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.829221:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e25ad lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201685.829241:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187605248:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201685.829775:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.840231:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187605248:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201685.840258:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e25ad lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc410f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.840271:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e25ad lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc410f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.840287:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e25ad lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc410f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.840298:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e25ad lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc410f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.840341:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e25ad lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc410f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.840350:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e25ad lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc410f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.840357:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e25ad lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc410f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201685.840369:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289538e00 x1875062187605248/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201685.840382:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e25ad lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc410f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.840395:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201685.840402:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201685.840428:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201685.840434:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e25ad 00000080:00010000:3.0:1788201685.840439:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201685.840442:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e25ad from it ffffaa180174bc10 00010000:00010000:3.0:1788201685.840446:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e25ad lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc410f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.840454:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e25ad lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc410f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.840480:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201685.840501:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-5, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201685.840506:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-5, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201685.840527:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-5,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201685.840578:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e25b4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.840585:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e25b4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.840594:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e25b4 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201685.840695:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187605376:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201685.840761:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.843813:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187605376:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201685.843839:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e25b4 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc412b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.843860:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e25b4 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xf6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc412b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.843873:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e25b4 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xf6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc412b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.843883:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e25b4 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc412b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.843893:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e25b4 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc412b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.843903:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e25b4 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xf6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc412b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.843910:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e25b4 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xf6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc412b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.843929:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e25b4 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc412b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201685.843941:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289538e00 x1875062187605376/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201685.843952:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e25b4 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc412b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201685.843965:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e25b4 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc412b expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.843979:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/246 (0 0) 00000080:00200000:3.0:1788201685.843986:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201685.844028:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xf6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201685.844039:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201685.844067:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xf6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201685.844121:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xf6:0x0](ffff8e8289e3d348): rc = 0 00000080:00200000:3.0:1788201685.844136:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xf6:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201685.844141:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xf6:0x0](ffff8e8289e3d348) for lock 0x9ae3109cee9e25b4 00000080:00010000:3.0:1788201685.844151:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xf6:0x0](ffff8e8289e3d348) 00000080:00010000:3.0:1788201685.844154:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e25b4 from it ffffaa180174be00 00010000:00010000:3.0:1788201685.844157:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e25b4 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc412b expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.844166:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e25b4 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc412b expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.844197:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201685.844213:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201685.844223:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e25ad lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc410f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.844233:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e25ad lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc410f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.844239:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201685.844244:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201685.844247:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e25ad 00000080:00010000:3.0:1788201685.844251:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e25ad from it ffffaa180174bd70 00010000:00010000:3.0:1788201685.844254:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e25ad lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc410f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.844260:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e25ad lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc410f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201685.844269:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201685.844275:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-5, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201685.844299:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e25ad lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc410f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201685.844362:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e25b4 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc412b expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.844373:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xf6:0x0]: losing layout lock 00010000:00010000:3.0:1788201685.844400:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e25b4 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xf6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc412b expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.844407:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e25ad lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc410f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201685.844413:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201685.844415:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e25b4 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xf6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc412b expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.844429:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e25ad lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc410f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201685.844452:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201685.844568:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e8289538e00 x1875062187605632/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201685.844592:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187605632:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201685.844737:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1000 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.849226:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187605632:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201685.849282:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xf6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201685.849294:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201685.849345:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xf6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201685.849353:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xf6:0x0](ffff8e8289e3d348) 00000080:00200000:3.0:1788201685.849440:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201685.849560:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.849688:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25bb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.849699:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25bb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.849708:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25bb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.849727:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187605760:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.849794:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.854979:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187605760:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.855017:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25bb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4155 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.855029:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25bb lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4155 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.855039:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25bb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4155 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.855049:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25bb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4155 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.855062:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25bb lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4155 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.855069:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25bb lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4155 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.855076:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25bb lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4155 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.855088:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d7100 x1875062187605760/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.855101:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25bb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4155 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.855114:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.855120:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.855148:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.855154:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e25bb 00000080:00010000:2.0:1788201685.855159:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.855163:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e25bb from it ffffaa180174bc10 00010000:00010000:2.0:1788201685.855166:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25bb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4155 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.855175:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25bb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4155 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.855199:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.855223:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-6, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.855229:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-6, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.855251:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-6,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.855290:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25c2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.855298:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25c2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.855714:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25c2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.855740:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187605888:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.855819:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.858828:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187605888:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.858854:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25c2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4171 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.858881:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25c2 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xf7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4171 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.858889:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25c2 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xf7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4171 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.858899:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25c2 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4171 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.858908:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25c2 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4171 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.858920:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25c2 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xf7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4171 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.858928:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25c2 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xf7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4171 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.858935:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25c2 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4171 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.858946:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d7100 x1875062187605888/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.858957:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25c2 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4171 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.858975:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25c2 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4171 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.858986:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/247 (0 0) 00000080:00200000:2.0:1788201685.858992:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.859100:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xf7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.859111:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.859168:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xf7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.859226:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xf7:0x0](ffff8e8289e3c2c8): rc = 0 00000080:00200000:2.0:1788201685.859241:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xf7:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.859246:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xf7:0x0](ffff8e8289e3c2c8) for lock 0x9ae3109cee9e25c2 00000080:00010000:2.0:1788201685.859256:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xf7:0x0](ffff8e8289e3c2c8) 00000080:00010000:2.0:1788201685.859259:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e25c2 from it ffffaa180174be00 00010000:00010000:2.0:1788201685.859262:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25c2 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc4171 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.859271:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25c2 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc4171 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.859302:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.859522:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.859681:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25bb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4155 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.859694:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25bb lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4155 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.859708:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.859713:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.859716:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e25bb 00000080:00010000:2.0:1788201685.859721:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e25bb from it ffffaa180174bd70 00010000:00010000:2.0:1788201685.859725:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25bb lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4155 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.859733:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25bb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4155 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.859743:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.859750:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-6, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.859776:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25bb lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc4155 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.859799:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25c2 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc4171 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.859810:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xf7:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.859840:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25c2 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xf7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc4171 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.859848:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25bb lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc4155 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.859854:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.859857:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25c2 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xf7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc4171 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.859874:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25bb lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc4155 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.859891:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.859901:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894d7100 x1875062187606144/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.859937:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187606144:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.860277:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.868052:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187606144:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.868115:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xf7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.868129:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.868158:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xf7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.868164:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xf7:0x0](ffff8e8289e3c2c8) 00000080:00200000:2.0:1788201685.868253:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.868273:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.868375:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25c9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.868385:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25c9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.868395:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25c9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.868413:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187606272:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.868498:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.870855:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187606272:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.870878:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25c9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc419b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.870890:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25c9 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc419b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.870902:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25c9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc419b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.870914:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25c9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc419b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.870935:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25c9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc419b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.870943:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25c9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc419b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.870951:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25c9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc419b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.870962:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d7800 x1875062187606272/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.870975:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25c9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc419b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.870991:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.871179:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.871204:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.871211:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e25c9 00000080:00010000:2.0:1788201685.871217:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.871220:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e25c9 from it ffffaa180174bc10 00010000:00010000:2.0:1788201685.871223:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25c9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc419b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.871231:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25c9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc419b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.871254:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.871277:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-7, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.871282:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-7, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.871302:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-7,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.871360:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25d0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.871368:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25d0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.871376:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25d0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.871393:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187606400:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.871454:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.875375:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187606400:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.875402:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25d0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc41b7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.875423:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25d0 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xf8:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc41b7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.875431:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25d0 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xf8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc41b7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.875441:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25d0 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc41b7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.875451:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25d0 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc41b7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.875463:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25d0 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xf8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc41b7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.875483:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25d0 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xf8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc41b7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.875490:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25d0 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc41b7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.875502:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d7800 x1875062187606400/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.875513:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25d0 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc41b7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.875526:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25d0 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc41b7 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.875536:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/248 (0 0) 00000080:00200000:2.0:1788201685.875543:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.875587:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xf8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.875599:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.875628:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xf8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.875697:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xf8:0x0](ffff8e8289e3ba88): rc = 0 00000080:00200000:2.0:1788201685.875712:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xf8:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.875718:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xf8:0x0](ffff8e8289e3ba88) for lock 0x9ae3109cee9e25d0 00000080:00010000:2.0:1788201685.875729:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xf8:0x0](ffff8e8289e3ba88) 00000080:00010000:2.0:1788201685.875733:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e25d0 from it ffffaa180174be00 00010000:00010000:2.0:1788201685.875736:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25d0 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc41b7 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.875745:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25d0 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc41b7 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.875775:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.875792:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.875803:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25c9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc419b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.875814:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25c9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc419b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.875825:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.875830:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.875833:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e25c9 00000080:00010000:2.0:1788201685.875837:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e25c9 from it ffffaa180174bd70 00010000:00010000:2.0:1788201685.875840:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25c9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc419b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.875848:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25c9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc419b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.875856:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.875863:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-7, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.875887:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25c9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc419b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.875909:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25d0 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc41b7 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.876256:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xf8:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.876291:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25d0 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xf8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc41b7 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.876300:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25c9 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc419b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.876366:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.876370:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25d0 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xf8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc41b7 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.876387:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25c9 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc419b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.876405:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.876414:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894d7800 x1875062187606656/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.876435:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187606656:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.876728:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.880851:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187606656:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.880913:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xf8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.880938:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.880963:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xf8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.880971:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xf8:0x0](ffff8e8289e3ba88) 00000080:00200000:2.0:1788201685.881075:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.881095:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.881147:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25d7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.881158:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25d7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.881168:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25d7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.881187:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187606784:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.881303:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.884388:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187606784:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.884411:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25d7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc41e1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.884423:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25d7 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc41e1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.884435:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25d7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc41e1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.884445:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25d7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc41e1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.884456:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25d7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc41e1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.884524:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25d7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc41e1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.884590:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25d7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc41e1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.884605:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d5180 x1875062187606784/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.884618:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25d7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc41e1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.884645:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.884652:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.884679:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.884684:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e25d7 00000080:00010000:2.0:1788201685.884689:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.884692:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e25d7 from it ffffaa180174bc10 00010000:00010000:2.0:1788201685.884695:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25d7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc41e1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.884703:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25d7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc41e1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.884727:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.884750:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-8, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.884755:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-8, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.884775:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-8,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.884811:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25de lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.884818:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25de lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.884827:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25de lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.884845:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187606912:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.885016:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.888187:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187606912:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.888210:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25de lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc41fd expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.888230:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25de lrc: 4/1,0 mode: --/PR res: [0x200006991:0xf9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc41fd expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.888238:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25de lrc: 4/1,0 mode: --/PR res: [0x200006991:0xf9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc41fd expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.888249:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25de lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc41fd expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.888258:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25de lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc41fd expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.888269:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25de lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xf9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc41fd expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.888277:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25de lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xf9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc41fd expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.888285:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25de lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xf9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc41fd expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.888295:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d5180 x1875062187606912/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.888384:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25de lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc41fd expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.888399:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25de lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc41fd expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.888409:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/249 (0 0) 00000080:00200000:2.0:1788201685.888415:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.888452:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xf9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.888515:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.888597:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xf9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.888685:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xf9:0x0](ffff8e8289e3a1c8): rc = 0 00000080:00200000:2.0:1788201685.888701:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xf9:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.888706:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xf9:0x0](ffff8e8289e3a1c8) for lock 0x9ae3109cee9e25de 00000080:00010000:2.0:1788201685.888717:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xf9:0x0](ffff8e8289e3a1c8) 00000080:00010000:2.0:1788201685.888720:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e25de from it ffffaa180174be00 00010000:00010000:2.0:1788201685.888724:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25de lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xf9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc41fd expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.888733:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25de lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc41fd expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.888762:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.888779:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.888788:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25d7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc41e1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.888798:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25d7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc41e1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.888809:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.888813:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.888817:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e25d7 00000080:00010000:2.0:1788201685.888821:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e25d7 from it ffffaa180174bd70 00010000:00010000:2.0:1788201685.888824:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25d7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc41e1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.888832:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25d7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc41e1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.888840:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.888847:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-8, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.888870:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25d7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc41e1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.888890:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25de lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xf9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc41fd expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.888898:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xf9:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.888922:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25de lrc: 1/0,0 mode: --/PR res: [0x200006991:0xf9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc41fd expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.888930:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25d7 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc41e1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.888935:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.888938:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25de lrc: 0/0,0 mode: --/PR res: [0x200006991:0xf9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc41fd expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.888952:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25d7 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc41e1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.888970:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.888977:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894d5180 x1875062187607168/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.889060:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d5180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187607168:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.889179:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.893411:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d5180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187607168:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.893592:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xf9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.893607:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.893640:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xf9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.893649:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xf9:0x0](ffff8e8289e3a1c8) 00000080:00200000:2.0:1788201685.893738:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.893758:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.893815:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25e5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.893825:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25e5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.893834:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25e5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.893853:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187607296:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.893966:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.896838:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187607296:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.896862:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25e5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4227 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.896875:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25e5 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4227 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.896886:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25e5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4227 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.896897:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25e5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4227 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.896909:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25e5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4227 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.896917:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25e5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4227 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.896924:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25e5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4227 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.896936:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d7b80 x1875062187607296/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.896950:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25e5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4227 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.896968:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.896975:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.897077:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.897086:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e25e5 00000080:00010000:2.0:1788201685.897092:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.897095:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e25e5 from it ffffaa180174bc10 00010000:00010000:2.0:1788201685.897099:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25e5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4227 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.897109:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25e5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4227 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.897133:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.897153:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-9, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.897158:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-9, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.897178:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-9,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.897212:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25ec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.897220:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25ec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.897228:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25ec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.897245:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187607424:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.897365:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.904061:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187607424:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.904091:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25ec lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4243 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.904114:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25ec lrc: 4/1,0 mode: --/PR res: [0x200006991:0xfa:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4243 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.904123:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25ec lrc: 4/1,0 mode: --/PR res: [0x200006991:0xfa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4243 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.904134:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25ec lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xfa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4243 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.904145:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25ec lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xfa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4243 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.904158:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25ec lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xfa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4243 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.904166:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25ec lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xfa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4243 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.904173:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25ec lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xfa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4243 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.904186:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894d7b80 x1875062187607424/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.904197:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25ec lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4243 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.904210:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25ec lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4243 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.904220:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/250 (0 0) 00000080:00200000:2.0:1788201685.904226:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.904265:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xfa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.904287:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.904342:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xfa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.904392:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xfa:0x0](ffff8e8289e3e3c8): rc = 0 00000080:00200000:2.0:1788201685.904405:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xfa:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.904410:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xfa:0x0](ffff8e8289e3e3c8) for lock 0x9ae3109cee9e25ec 00000080:00010000:2.0:1788201685.904419:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xfa:0x0](ffff8e8289e3e3c8) 00000080:00010000:2.0:1788201685.904422:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e25ec from it ffffaa180174be00 00010000:00010000:2.0:1788201685.904425:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25ec lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc4243 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.904434:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25ec lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xfa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc4243 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.904756:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.904775:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.904785:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25e5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4227 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.904796:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25e5 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4227 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.904809:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.904813:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.904816:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e25e5 00000080:00010000:2.0:1788201685.904820:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e25e5 from it ffffaa180174bd70 00010000:00010000:2.0:1788201685.904823:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25e5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4227 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.904831:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25e5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4227 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.904840:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.904847:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-9, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.904870:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25e5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc4227 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.904891:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25ec lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xfa:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc4243 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.904900:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xfa:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.904927:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25ec lrc: 1/0,0 mode: --/PR res: [0x200006991:0xfa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc4243 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.904935:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25e5 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc4227 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.904940:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.904943:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e25ec lrc: 0/0,0 mode: --/PR res: [0x200006991:0xfa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc4243 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.904957:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e25e5 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc4227 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.904974:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.904982:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894d7b80 x1875062187607680/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.905015:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894d7b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187607680:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.905092:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.908732:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894d7b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187607680:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.908782:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xfa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.908792:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.908816:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xfa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.908821:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xfa:0x0](ffff8e8289e3e3c8) 00000080:00200000:2.0:1788201685.908893:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.908909:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.908963:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25f3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.908972:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25f3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.908980:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25f3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.908998:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187607808:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.909073:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.910845:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187607808:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.910862:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25f3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc426d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.910870:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25f3 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc426d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.910878:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25f3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc426d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.910885:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25f3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc426d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.910892:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25f3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc426d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.910897:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25f3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc426d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.910902:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25f3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc426d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.910909:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910cb100 x1875062187607808/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.910918:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25f3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc426d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.910935:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.910941:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.910965:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.910972:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e25f3 00000080:00010000:2.0:1788201685.910976:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.910979:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e25f3 from it ffffaa180174bc10 00010000:00010000:2.0:1788201685.910983:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25f3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc426d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.910991:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25f3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc426d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.911013:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.911033:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-10, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.911037:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-10, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.911058:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-10,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.911091:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25fa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.911098:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25fa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.911106:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25fa lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.911122:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187607936:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.911178:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.912962:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187607936:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.912982:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25fa lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4289 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.913012:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25fa lrc: 4/1,0 mode: --/PR res: [0x200006991:0xfb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4289 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.913021:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25fa lrc: 4/1,0 mode: --/PR res: [0x200006991:0xfb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4289 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.913029:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25fa lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xfb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4289 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.913037:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25fa lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xfb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4289 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.913045:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25fa lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xfb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4289 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.913050:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25fa lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xfb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4289 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.913055:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25fa lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xfb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4289 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.913063:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910cb100 x1875062187607936/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.913069:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25fa lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4289 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.913078:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25fa lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4289 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.913085:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/251 (0 0) 00000080:00200000:2.0:1788201685.913090:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.913123:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xfb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.913132:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.913153:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xfb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.913195:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xfb:0x0](ffff8e8289e3f448): rc = 0 00000080:00200000:2.0:1788201685.913205:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xfb:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.913209:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xfb:0x0](ffff8e8289e3f448) for lock 0x9ae3109cee9e25fa 00000080:00010000:2.0:1788201685.913216:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xfb:0x0](ffff8e8289e3f448) 00000080:00010000:2.0:1788201685.913219:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e25fa from it ffffaa180174be00 00010000:00010000:2.0:1788201685.913221:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25fa lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc4289 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.913227:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25fa lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xfb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc4289 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.913247:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.913257:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.913263:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25f3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc426d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.913270:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25f3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc426d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.913278:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.913280:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.913282:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e25f3 00000080:00010000:2.0:1788201685.913284:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e25f3 from it ffffaa180174bd70 00010000:00010000:2.0:1788201685.913286:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25f3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc426d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.913290:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25f3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc426d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.913294:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.913298:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-10, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.913341:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25f3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc426d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.913357:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25fa lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xfb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc4289 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.913363:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xfb:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.913384:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25fa lrc: 1/0,0 mode: --/PR res: [0x200006991:0xfb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc4289 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.913389:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25f3 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc426d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.913392:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.913394:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e25fa lrc: 0/0,0 mode: --/PR res: [0x200006991:0xfb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc4289 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.913404:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e25f3 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc426d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.913416:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.913421:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82910cb100 x1875062187608192/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.913436:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187608192:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.913494:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.916057:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb100 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187608192:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.916101:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xfb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.916111:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.916130:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xfb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.916135:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xfb:0x0](ffff8e8289e3f448) 00000080:00200000:2.0:1788201685.916242:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.916256:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.916294:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2601 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.916300:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2601 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.916329:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2601 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.916344:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910ca300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187608320:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.916389:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.918112:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910ca300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187608320:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.918129:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2601 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42b3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.918136:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2601 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42b3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.918144:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2601 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc42b3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.918151:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2601 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42b3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.918159:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2601 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc42b3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.918164:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2601 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc42b3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.918168:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2601 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc42b3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.918175:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910ca300 x1875062187608320/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.918184:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2601 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42b3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.918196:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.918200:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.918222:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.918228:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2601 00000080:00010000:2.0:1788201685.918234:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.918237:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2601 from it ffffaa180174bc10 00010000:00010000:2.0:1788201685.918240:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2601 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc42b3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.918248:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2601 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42b3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.918268:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.918286:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-11, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.918291:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-11, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.918341:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-11,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.918377:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2608 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.918383:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2608 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.918391:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2608 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.918407:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910ca300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187608448:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.918473:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.920857:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910ca300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187608448:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.920875:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2608 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42cf expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.920890:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2608 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xfc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42cf expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.920895:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2608 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xfc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42cf expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.920902:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2608 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xfc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc42cf expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.920908:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2608 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xfc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42cf expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.920914:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2608 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xfc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc42cf expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.920919:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2608 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xfc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc42cf expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.920924:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2608 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xfc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc42cf expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.920930:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910ca300 x1875062187608448/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.920938:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2608 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42cf expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.920946:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2608 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42cf expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.920953:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/252 (0 0) 00000080:00200000:2.0:1788201685.920958:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.921010:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xfc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.921022:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.921045:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xfc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.921093:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xfc:0x0](ffff8e8289e38908): rc = 0 00000080:00200000:2.0:1788201685.921105:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xfc:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.921109:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xfc:0x0](ffff8e8289e38908) for lock 0x9ae3109cee9e2608 00000080:00010000:2.0:1788201685.921116:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xfc:0x0](ffff8e8289e38908) 00000080:00010000:2.0:1788201685.921119:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2608 from it ffffaa180174be00 00010000:00010000:2.0:1788201685.921121:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2608 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc42cf expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.921127:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2608 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xfc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc42cf expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.921150:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.921162:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.921167:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2601 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42b3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.921174:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2601 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42b3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.921182:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.921185:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.921187:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2601 00000080:00010000:2.0:1788201685.921189:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2601 from it ffffaa180174bd70 00010000:00010000:2.0:1788201685.921191:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2601 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc42b3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.921195:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2601 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42b3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.921199:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.921203:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-11, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.921220:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2601 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc42b3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.921234:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2608 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xfc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc42cf expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.921240:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xfc:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.921264:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2608 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xfc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc42cf expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.921270:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2601 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc42b3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.921273:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.921275:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2608 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xfc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc42cf expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.921289:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2601 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc42b3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.921302:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.921371:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82910ca300 x1875062187608704/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.921390:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910ca300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187608704:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.921440:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.924161:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910ca300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187608704:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.924205:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xfc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.924215:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.924236:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xfc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.924241:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xfc:0x0](ffff8e8289e38908) 00000080:00200000:2.0:1788201685.924330:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.924345:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.924382:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e260f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.924388:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e260f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.924395:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e260f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.924406:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187608832:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.924463:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.926833:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187608832:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.926852:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e260f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42f9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.926860:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e260f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42f9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.926868:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e260f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc42f9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.926874:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e260f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42f9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.926882:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e260f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc42f9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.926887:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e260f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc42f9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.926891:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e260f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc42f9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.926898:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910cb800 x1875062187608832/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.926907:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e260f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42f9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.926920:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.926927:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.926951:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.926957:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e260f 00000080:00010000:2.0:1788201685.926962:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.926965:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e260f from it ffffaa180174bc10 00010000:00010000:2.0:1788201685.926969:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e260f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc42f9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.926976:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e260f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42f9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.927011:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.927033:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-12, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.927037:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-12, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.927057:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-12,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.927089:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2616 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.927094:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2616 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.927100:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2616 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.927112:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187608960:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.927162:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.929118:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187608960:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.929137:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2616 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4315 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.929152:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2616 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4315 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.929157:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2616 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4315 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.929164:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2616 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4315 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.929170:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2616 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4315 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.929178:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2616 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4315 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.929182:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2616 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4315 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.929187:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2616 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4315 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.929194:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910cb800 x1875062187608960/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.929201:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2616 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4315 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.929209:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2616 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4315 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.929215:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/253 (0 0) 00000080:00200000:2.0:1788201685.929219:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.929252:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xfd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.929260:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.929283:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xfd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.929347:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xfd:0x0](ffff8e8289e5ec08): rc = 0 00000080:00200000:2.0:1788201685.929358:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xfd:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.929362:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xfd:0x0](ffff8e8289e5ec08) for lock 0x9ae3109cee9e2616 00000080:00010000:2.0:1788201685.929369:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xfd:0x0](ffff8e8289e5ec08) 00000080:00010000:2.0:1788201685.929371:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2616 from it ffffaa180174be00 00010000:00010000:2.0:1788201685.929373:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2616 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc4315 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.929379:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2616 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc4315 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.929401:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.929412:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.929418:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e260f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42f9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.929424:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e260f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42f9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.929432:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.929435:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.929437:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e260f 00000080:00010000:2.0:1788201685.929438:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e260f from it ffffaa180174bd70 00010000:00010000:2.0:1788201685.929440:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e260f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc42f9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.929445:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e260f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc42f9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.929450:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.929454:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-12, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.929657:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e260f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc42f9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.929678:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2616 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc4315 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.929686:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xfd:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.929716:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2616 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc4315 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.929723:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e260f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc42f9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.929726:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.929728:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2616 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xfd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc4315 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.929741:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e260f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc42f9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.929754:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.929760:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82910cb800 x1875062187609216/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.929775:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187609216:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.929861:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.932670:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187609216:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.932727:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xfd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.932737:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.932759:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xfd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.932765:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xfd:0x0](ffff8e8289e5ec08) 00000080:00200000:2.0:1788201685.932836:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.932851:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.932893:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e261d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.932899:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e261d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.932905:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e261d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.932918:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187609344:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.933148:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.935163:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187609344:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.935187:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e261d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc433f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.935200:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e261d lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc433f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.935210:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e261d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc433f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.935220:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e261d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc433f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.935230:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e261d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc433f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.935238:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e261d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc433f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.935244:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e261d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc433f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.935256:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910c9500 x1875062187609344/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.935269:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e261d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc433f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.935285:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.935292:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.935510:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.935517:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e261d 00000080:00010000:2.0:1788201685.935522:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.935524:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e261d from it ffffaa180174bc10 00010000:00010000:2.0:1788201685.935526:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e261d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc433f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.935575:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e261d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc433f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.935595:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.935615:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-13, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.935620:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-13, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.935650:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-13,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.935687:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2624 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.935694:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2624 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.935702:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2624 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.935720:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187609472:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.935807:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.938646:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187609472:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.938671:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2624 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc435b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.938691:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2624 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc435b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.938699:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2624 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc435b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.938709:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2624 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc435b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.938719:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2624 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc435b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.938732:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2624 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc435b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.938737:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2624 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc435b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.938742:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2624 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc435b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.938750:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910c9500 x1875062187609472/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.938758:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2624 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc435b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.938768:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2624 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc435b expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.938775:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/254 (0 0) 00000080:00200000:2.0:1788201685.938780:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.938817:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xfe:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.938826:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.938849:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xfe:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.938894:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xfe:0x0](ffff8e8289e59148): rc = 0 00000080:00200000:2.0:1788201685.938905:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xfe:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.938909:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xfe:0x0](ffff8e8289e59148) for lock 0x9ae3109cee9e2624 00000080:00010000:2.0:1788201685.938917:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xfe:0x0](ffff8e8289e59148) 00000080:00010000:2.0:1788201685.938920:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2624 from it ffffaa180174be00 00010000:00010000:2.0:1788201685.938922:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2624 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc435b expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.938928:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2624 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc435b expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.938951:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.938965:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.938973:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e261d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc433f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.938984:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e261d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc433f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.939132:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.939137:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.939139:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e261d 00000080:00010000:2.0:1788201685.939143:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e261d from it ffffaa180174bd70 00010000:00010000:2.0:1788201685.939145:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e261d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc433f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.939150:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e261d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc433f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.939157:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.939162:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-13, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.939182:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e261d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc433f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.939198:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2624 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc435b expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.939204:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xfe:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.939225:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2624 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc435b expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.939229:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e261d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc433f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.939233:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.939235:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e2624 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xfe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc435b expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.939246:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e261d lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc433f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.939259:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.939264:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82910c9500 x1875062187609728/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.939280:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187609728:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.939522:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.942119:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c9500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187609728:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.942169:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xfe:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.942181:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.942205:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xfe:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.942212:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xfe:0x0](ffff8e8289e59148) 00000080:00200000:2.0:1788201685.942282:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.942297:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.942373:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e262b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.942379:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e262b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.942386:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e262b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.942399:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c8a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187609856:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.942447:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.944136:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c8a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187609856:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.944155:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e262b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4385 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.944164:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e262b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4385 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.944172:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e262b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4385 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.944178:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e262b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4385 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.944186:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e262b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4385 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.944191:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e262b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4385 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.944195:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e262b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4385 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.944202:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910c8a80 x1875062187609856/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.944211:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e262b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4385 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.944223:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.944227:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.944246:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.944251:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e262b 00000080:00010000:2.0:1788201685.944255:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.944257:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e262b from it ffffaa180174bc10 00010000:00010000:2.0:1788201685.944259:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e262b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4385 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.944264:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e262b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4385 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.944280:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.944296:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-14, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.944299:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-14, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.944335:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-14,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.944358:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2632 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.944363:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2632 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.944368:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2632 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.944378:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c8a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187609984:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.944416:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.946872:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c8a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187609984:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.946896:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2632 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc43a1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.946915:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2632 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xff:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc43a1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.946922:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2632 lrc: 4/1,0 mode: --/PR res: [0x200006991:0xff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc43a1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.946932:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2632 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc43a1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.946941:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2632 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc43a1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.946953:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2632 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc43a1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.946961:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2632 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0xff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc43a1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.946969:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2632 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0xff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc43a1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.946981:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910c8a80 x1875062187609984/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.946991:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2632 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc43a1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.947020:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2632 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc43a1 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.947029:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/255 (0 0) 00000080:00200000:2.0:1788201685.947035:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.947076:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xff:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.947086:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.947113:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xff:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.947170:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0xff:0x0](ffff8e8289e3ec08): rc = 0 00000080:00200000:2.0:1788201685.947186:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0xff:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.947192:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0xff:0x0](ffff8e8289e3ec08) for lock 0x9ae3109cee9e2632 00000080:00010000:2.0:1788201685.947201:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0xff:0x0](ffff8e8289e3ec08) 00000080:00010000:2.0:1788201685.947205:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2632 from it ffffaa180174be00 00010000:00010000:2.0:1788201685.947208:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2632 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0xff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc43a1 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.947217:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2632 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc43a1 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.947244:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.947262:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.947271:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e262b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4385 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.947281:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e262b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4385 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.947291:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.947295:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.947298:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e262b 00000080:00010000:2.0:1788201685.947302:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e262b from it ffffaa180174bd70 00010000:00010000:2.0:1788201685.947304:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e262b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4385 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.947350:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e262b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4385 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.947359:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.947366:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-14, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.947395:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e262b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc4385 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.947418:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2632 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0xff:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc43a1 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.947426:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0xff:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.947454:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2632 lrc: 1/0,0 mode: --/PR res: [0x200006991:0xff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc43a1 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.947473:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e262b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc4385 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.947479:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.947482:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2632 lrc: 0/0,0 mode: --/PR res: [0x200006991:0xff:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc43a1 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.947498:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e262b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc4385 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.947516:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.947523:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82910c8a80 x1875062187610240/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.947545:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c8a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187610240:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.947615:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.951641:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c8a80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187610240:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.951702:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0xff:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.951715:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.951742:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0xff:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.951749:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0xff:0x0](ffff8e8289e3ec08) 00000080:00200000:2.0:1788201685.951840:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.951863:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.951923:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2639 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.951934:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2639 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.951945:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2639 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.951966:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187610368:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.952059:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.955116:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187610368:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.955143:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2639 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc43cb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.955155:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2639 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc43cb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.955166:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2639 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc43cb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.955178:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2639 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc43cb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.955191:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2639 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc43cb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.955199:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2639 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc43cb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.955207:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2639 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc43cb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.955219:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910cb480 x1875062187610368/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.955232:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2639 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc43cb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.955248:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.955255:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.955295:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.955302:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2639 00000080:00010000:2.0:1788201685.955338:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.955341:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2639 from it ffffaa180174bc10 00010000:00010000:2.0:1788201685.955345:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2639 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc43cb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.955352:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2639 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc43cb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.955376:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.955399:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-15, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.955404:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-15, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.955426:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-15,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.955477:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2640 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.955485:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2640 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.955494:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2640 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.955513:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187610496:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.955593:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.959160:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187610496:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.959184:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2640 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc43e7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.959204:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2640 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x100:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc43e7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.959213:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2640 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x100:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc43e7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.959224:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2640 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x100:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc43e7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.959234:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2640 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x100:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc43e7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.959247:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2640 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x100:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc43e7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.959255:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2640 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x100:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc43e7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.959261:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2640 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x100:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc43e7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.959273:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910cb480 x1875062187610496/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.959283:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2640 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x100:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc43e7 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.959296:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2640 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x100:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc43e7 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.959305:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/256 (0 0) 00000080:00200000:2.0:1788201685.959342:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.959384:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x100:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.959394:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.959422:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x100:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.959532:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x100:0x0](ffff8e8289e39148): rc = 0 00000080:00200000:2.0:1788201685.959558:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x100:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.959564:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x100:0x0](ffff8e8289e39148) for lock 0x9ae3109cee9e2640 00000080:00010000:2.0:1788201685.959574:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x100:0x0](ffff8e8289e39148) 00000080:00010000:2.0:1788201685.959578:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2640 from it ffffaa180174be00 00010000:00010000:2.0:1788201685.959581:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2640 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x100:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc43e7 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.959590:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2640 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x100:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc43e7 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.959620:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.959645:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.959655:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2639 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc43cb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.959666:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2639 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc43cb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.959677:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.959682:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.959685:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2639 00000080:00010000:2.0:1788201685.959688:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2639 from it ffffaa180174bd70 00010000:00010000:2.0:1788201685.959691:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2639 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc43cb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.959699:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2639 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc43cb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.959708:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.959715:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-15, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.959740:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2639 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc43cb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.959762:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2640 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x100:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc43e7 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.959771:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x100:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.959800:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2640 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x100:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc43e7 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.959807:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2639 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc43cb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.959813:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.959815:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2640 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x100:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc43e7 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.959831:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e2639 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc43cb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.959848:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.959856:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82910cb480 x1875062187610752/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.959880:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187610752:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.959957:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.963510:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cb480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187610752:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.963578:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x100:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.963590:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.963614:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x100:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.963621:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x100:0x0](ffff8e8289e39148) 00000080:00200000:2.0:1788201685.963715:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.963735:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.963791:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2647 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.963801:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2647 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.963810:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2647 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.963827:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c8e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187610880:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.963904:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.966342:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c8e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187610880:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.966366:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2647 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4411 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.966378:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2647 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4411 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.966389:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2647 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4411 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.966401:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2647 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4411 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.966414:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2647 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4411 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.966422:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2647 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4411 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.966429:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2647 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4411 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.966441:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910c8e00 x1875062187610880/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.966454:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2647 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4411 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.966485:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.966492:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.966516:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.966522:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2647 00000080:00010000:2.0:1788201685.966527:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.966598:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2647 from it ffffaa180174bc10 00010000:00010000:2.0:1788201685.966604:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2647 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4411 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.966613:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2647 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4411 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.966645:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.966666:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-16, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.966671:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-16, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.966692:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-16,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.966729:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e264e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.966737:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e264e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.966746:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e264e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.966764:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c8e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187611008:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.966835:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.969728:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c8e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187611008:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.969753:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e264e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc442d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.969775:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e264e lrc: 4/1,0 mode: --/PR res: [0x200006991:0x101:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc442d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.969783:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e264e lrc: 4/1,0 mode: --/PR res: [0x200006991:0x101:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc442d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.969794:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e264e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x101:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc442d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.969804:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e264e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x101:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc442d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.969817:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e264e lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x101:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc442d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.969825:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e264e lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x101:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc442d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.969831:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e264e lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x101:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc442d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.969842:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910c8e00 x1875062187611008/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.969852:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e264e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x101:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc442d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.969865:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e264e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x101:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc442d expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.969873:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/257 (0 0) 00000080:00200000:2.0:1788201685.969879:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.969917:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x101:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.969928:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.969955:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x101:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.970132:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x101:0x0](ffff8e8289e3b248): rc = 0 00000080:00200000:2.0:1788201685.970149:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x101:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.970154:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x101:0x0](ffff8e8289e3b248) for lock 0x9ae3109cee9e264e 00000080:00010000:2.0:1788201685.970163:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x101:0x0](ffff8e8289e3b248) 00000080:00010000:2.0:1788201685.970167:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e264e from it ffffaa180174be00 00010000:00010000:2.0:1788201685.970171:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e264e lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x101:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc442d expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.970180:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e264e lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x101:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc442d expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.970210:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.970226:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.970236:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2647 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4411 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.970247:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2647 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4411 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.970258:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.970262:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.970265:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2647 00000080:00010000:2.0:1788201685.970269:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2647 from it ffffaa180174bd70 00010000:00010000:2.0:1788201685.970271:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2647 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4411 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.970279:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2647 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4411 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.970287:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.970295:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-16, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.970396:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2647 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc4411 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.970419:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e264e lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x101:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc442d expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.970428:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x101:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.970474:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e264e lrc: 1/0,0 mode: --/PR res: [0x200006991:0x101:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc442d expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.970483:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2647 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc4411 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.970489:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.970492:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e264e lrc: 0/0,0 mode: --/PR res: [0x200006991:0x101:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc442d expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.970508:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2647 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc4411 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.970524:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.970541:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82910c8e00 x1875062187611264/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.970563:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910c8e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187611264:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.970639:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.974165:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910c8e00 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187611264:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.974220:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x101:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.974234:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.974258:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x101:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.974265:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x101:0x0](ffff8e8289e3b248) 00000080:00200000:2.0:1788201685.974381:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.974400:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.974448:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2655 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.974467:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2655 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.974477:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2655 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.974494:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187611392:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.974551:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.977036:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187611392:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.977060:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2655 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4457 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.977071:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2655 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4457 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.977083:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2655 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4457 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.977094:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2655 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4457 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.977107:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2655 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4457 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.977116:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2655 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4457 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.977123:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2655 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4457 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.977134:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910cbb80 x1875062187611392/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.977147:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2655 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4457 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.977165:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.977172:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.977197:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.977204:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2655 00000080:00010000:2.0:1788201685.977209:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.977212:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2655 from it ffffaa180174bc10 00010000:00010000:2.0:1788201685.977216:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2655 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4457 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.977224:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2655 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4457 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.977246:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.977267:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-17, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.977271:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-17, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.977290:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-17,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201685.977344:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e265c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.977352:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e265c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.977361:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e265c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.977378:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187611520:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.977438:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.980163:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187611520:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.980188:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e265c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4473 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.980208:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e265c lrc: 4/1,0 mode: --/PR res: [0x200006991:0x102:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4473 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.980215:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e265c lrc: 4/1,0 mode: --/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4473 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.980225:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e265c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4473 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.980235:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e265c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4473 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.980246:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e265c lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4473 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.980254:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e265c lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4473 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.980260:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e265c lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4473 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.980270:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82910cbb80 x1875062187611520/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.980280:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e265c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4473 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201685.980292:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e265c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4473 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.980300:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/258 (0 0) 00000080:00200000:2.0:1788201685.980350:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201685.980395:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x102:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.980408:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.980435:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x102:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.980506:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x102:0x0](ffff8e8289e5b248): rc = 0 00000080:00200000:2.0:1788201685.980523:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x102:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201685.980529:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x102:0x0](ffff8e8289e5b248) for lock 0x9ae3109cee9e265c 00000080:00010000:2.0:1788201685.980550:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x102:0x0](ffff8e8289e5b248) 00000080:00010000:2.0:1788201685.980553:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e265c from it ffffaa180174be00 00010000:00010000:2.0:1788201685.980557:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e265c lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc4473 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.980566:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e265c lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc4473 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.980599:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.980618:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.980628:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2655 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4457 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.980649:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2655 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4457 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.980660:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201685.980665:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.980667:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2655 00000080:00010000:2.0:1788201685.980671:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2655 from it ffffaa180174bd70 00010000:00010000:2.0:1788201685.980674:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2655 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4457 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.980681:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2655 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4457 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201685.980690:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201685.980697:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-17, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201685.980723:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2655 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc4457 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.980743:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e265c lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc4473 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.980752:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x102:0x0]: losing layout lock 00010000:00010000:2.0:1788201685.980785:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e265c lrc: 1/0,0 mode: --/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc4473 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.980794:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2655 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc4457 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201685.980799:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201685.980802:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e265c lrc: 0/0,0 mode: --/PR res: [0x200006991:0x102:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc4473 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201685.980820:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e2655 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc4457 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201685.980837:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201685.980844:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82910cbb80 x1875062187611776/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201685.980866:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187611776:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201685.980938:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.984737:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82910cbb80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187611776:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201685.984803:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x102:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201685.984816:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201685.984843:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x102:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201685.984852:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x102:0x0](ffff8e8289e5b248) 00000080:00200000:2.0:1788201685.984942:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201685.984965:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201685.985175:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2663 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.985185:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2663 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.985194:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2663 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201685.985212:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187611904:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201685.985285:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201685.987901:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187611904:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201685.987925:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2663 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc449d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.987938:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2663 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc449d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.987949:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2663 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc449d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.987960:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2663 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc449d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.987972:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2663 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc449d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.987979:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2663 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc449d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.987985:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2663 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc449d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201685.988164:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894e5500 x1875062187611904/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201685.988181:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2663 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc449d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.988200:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201685.988206:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201685.988425:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201685.988434:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2663 00000080:00010000:2.0:1788201685.988439:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201685.988442:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2663 from it ffffaa180174bc10 00010000:00010000:2.0:1788201685.988446:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2663 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc449d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201685.988455:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2663 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc449d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201685.988478:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201685.988501:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-18, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201685.988505:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-18, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201685.988680:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-18,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201685.988824:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e266a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.988832:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e266a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.988841:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e266a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201685.988862:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187612032:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201685.988980:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.992168:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187612032:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201685.992197:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e266a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc44b9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.992220:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e266a lrc: 4/1,0 mode: --/PR res: [0x200006991:0x103:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc44b9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.992228:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e266a lrc: 4/1,0 mode: --/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc44b9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.992238:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e266a lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc44b9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.992248:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e266a lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc44b9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.992260:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e266a lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc44b9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.992268:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e266a lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc44b9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.992276:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e266a lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc44b9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201685.992287:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828953b800 x1875062187612032/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201685.992299:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e266a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc44b9 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201685.992422:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e266a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc44b9 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.992433:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/259 (0 0) 00000080:00200000:3.0:1788201685.992440:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201685.992585:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x103:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201685.992596:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201685.992623:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x103:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201685.992691:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x103:0x0](ffff8e8289e5aa08): rc = 0 00000080:00200000:3.0:1788201685.992706:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x103:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201685.992712:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x103:0x0](ffff8e8289e5aa08) for lock 0x9ae3109cee9e266a 00000080:00010000:3.0:1788201685.992721:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x103:0x0](ffff8e8289e5aa08) 00000080:00010000:3.0:1788201685.992725:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e266a from it ffffaa180174be00 00010000:00010000:3.0:1788201685.992728:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e266a lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc44b9 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.992738:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e266a lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc44b9 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.992770:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201685.992788:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201685.992799:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2663 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc449d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.992809:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2663 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc449d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.992821:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201685.992826:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201685.992829:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2663 00000080:00010000:3.0:1788201685.992833:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2663 from it ffffaa180174bd70 00010000:00010000:3.0:1788201685.992836:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2663 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc449d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.992844:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2663 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc449d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201685.992853:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201685.992860:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-18, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201685.992885:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2663 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc449d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201685.992907:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e266a lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc44b9 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201685.992917:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x103:0x0]: losing layout lock 00010000:00010000:3.0:1788201685.992947:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e266a lrc: 1/0,0 mode: --/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc44b9 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.992955:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2663 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc449d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201685.992961:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201685.992963:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e266a lrc: 0/0,0 mode: --/PR res: [0x200006991:0x103:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc44b9 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201685.992979:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463200/0x9ae3109cee9e2663 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc449d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201685.993175:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201685.993185:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828953b800 x1875062187612288/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201685.993207:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187612288:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201685.993702:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1000 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.996983:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b800 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187612288:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201685.997058:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x103:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201685.997072:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201685.997097:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x103:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201685.997104:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x103:0x0](ffff8e8289e5aa08) 00000080:00200000:3.0:1788201685.997190:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201685.997210:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201685.997263:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2671 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.997273:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2671 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.997283:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2671 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201685.997301:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187612416:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201685.997402:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201685.999888:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187612416:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201685.999911:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2671 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc44e3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.999920:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2671 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc44e3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.999932:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2671 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc44e3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.999942:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2671 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc44e3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.999953:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2671 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc44e3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.999961:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2671 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc44e3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201685.999969:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2671 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc44e3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201685.999980:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828953a300 x1875062187612416/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201685.999993:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2671 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc44e3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.000022:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201686.000029:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201686.000053:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201686.000059:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2671 00000080:00010000:3.0:1788201686.000065:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201686.000068:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2671 from it ffffaa180174bc10 00010000:00010000:3.0:1788201686.000071:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2671 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc44e3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.000080:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2671 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc44e3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201686.000103:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201686.000123:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-19, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201686.000126:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-19, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201686.000144:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-19,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201686.000175:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2678 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.000180:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2678 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.000186:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2678 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201686.000200:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187612544:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201686.000257:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201686.002273:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187612544:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201686.002292:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2678 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc44ff expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.002334:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2678 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x104:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc44ff expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.002340:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2678 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x104:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc44ff expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.002347:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2678 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x104:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc44ff expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.002353:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2678 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x104:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc44ff expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.002362:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2678 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x104:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc44ff expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.002367:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2678 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x104:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc44ff expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.002372:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2678 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x104:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc44ff expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201686.002379:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828953a300 x1875062187612544/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201701 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201686.002387:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2678 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x104:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc44ff expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201686.002396:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2678 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x104:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc44ff expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.002402:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/260 (0 0) 00000080:00200000:3.0:1788201686.002407:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201686.002443:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x104:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201686.002451:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201686.002650:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x104:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201686.002709:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x104:0x0](ffff8e8289e3aa08): rc = 0 00000080:00200000:3.0:1788201686.002725:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x104:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201686.002730:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x104:0x0](ffff8e8289e3aa08) for lock 0x9ae3109cee9e2678 00000080:00010000:3.0:1788201686.002738:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x104:0x0](ffff8e8289e3aa08) 00000080:00010000:3.0:1788201686.002740:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2678 from it ffffaa180174be00 00010000:00010000:3.0:1788201686.002743:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2678 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x104:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc44ff expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.002750:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2678 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x104:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc44ff expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201686.002772:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201686.002785:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201686.002791:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2671 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc44e3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201686.002800:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2671 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc44e3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201686.002807:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201686.002810:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201686.002812:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e2671 00000080:00010000:3.0:1788201686.002814:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2671 from it ffffaa180174bd70 00010000:00010000:3.0:1788201686.002816:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2671 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc44e3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201686.002820:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2671 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc44e3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201686.002824:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201686.002830:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-19, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201686.002848:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2671 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc44e3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201686.002863:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2678 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x104:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc44ff expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201686.002869:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x104:0x0]: losing layout lock 00010000:00010000:3.0:1788201686.002891:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2678 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x104:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc44ff expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201686.002896:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2671 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc44e3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201686.002899:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201686.002901:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2678 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x104:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc44ff expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201686.002912:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e2671 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc44e3 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201686.002928:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201686.002936:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828953a300 x1875062187612800/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201686.002958:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187612800:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201686.003148:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1000 going to sleep for 11 seconds 00000100:00100000:3.0:1788201686.006264:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a300 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187612800:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201686.006367:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x104:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201686.006378:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201686.006401:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x104:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201686.006407:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x104:0x0](ffff8e8289e3aa08) 00000080:00200000:3.0:1788201686.006579:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201686.006597:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201686.006654:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e267f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.006664:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e267f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.006674:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e267f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201686.006691:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187612928:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201686.006757:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201686.008818:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187612928:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201686.008838:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e267f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4529 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.008847:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e267f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4529 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.008856:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e267f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4529 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.008863:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e267f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4529 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.008871:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e267f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4529 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.008877:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e267f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4529 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.008881:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e267f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4529 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201686.008888:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828953b480 x1875062187612928/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201702 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201686.008897:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e267f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4529 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.008910:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201686.008914:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201686.008937:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201686.008943:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e267f 00000080:00010000:3.0:1788201686.008948:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201686.008951:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e267f from it ffffaa180174bc10 00010000:00010000:3.0:1788201686.008954:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e267f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4529 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.008963:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e267f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4529 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201686.008984:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201686.009019:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-20, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201686.009024:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-20, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201686.009041:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-20,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201686.009075:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2686 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.009083:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2686 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.009091:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2686 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201686.009110:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187613056:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201686.009194:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201686.011612:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187613056:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201686.011646:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2686 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4545 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.011666:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2686 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x105:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4545 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.011674:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2686 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x105:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4545 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.011684:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2686 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x105:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4545 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.011695:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2686 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x105:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4545 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.011706:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2686 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x105:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4545 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.011715:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2686 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x105:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4545 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.011722:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2686 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x105:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4545 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201686.011731:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e828953b480 x1875062187613056/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201702 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201686.011739:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2686 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x105:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4545 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201686.011748:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2686 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x105:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4545 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.011755:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/261 (0 0) 00000080:00200000:3.0:1788201686.011760:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201686.011795:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x105:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201686.011803:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201686.011825:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x105:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201686.011869:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x105:0x0](ffff8e8289e380c8): rc = 0 00000080:00200000:3.0:1788201686.011879:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x105:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201686.011883:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x105:0x0](ffff8e8289e380c8) for lock 0x9ae3109cee9e2686 00000080:00010000:3.0:1788201686.011890:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x105:0x0](ffff8e8289e380c8) 00000080:00010000:3.0:1788201686.011892:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2686 from it ffffaa180174be00 00010000:00010000:3.0:1788201686.011894:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2686 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x105:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc4545 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.011901:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2686 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x105:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc4545 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201686.011924:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201686.011939:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201686.011948:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e267f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4529 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201686.011958:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e267f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4529 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201686.011969:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201686.011973:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201686.011976:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e267f 00000080:00010000:3.0:1788201686.011980:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e267f from it ffffaa180174bd70 00010000:00010000:3.0:1788201686.011983:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e267f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4529 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201686.011990:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e267f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4529 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201686.012085:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201686.012091:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-20, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:3.0:1788201686.012113:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e267f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc4529 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201686.012129:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2686 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x105:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc4545 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201686.012134:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x105:0x0]: losing layout lock 00010000:00010000:3.0:1788201686.012156:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2686 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x105:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc4545 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201686.012161:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e267f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc4529 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788201686.012164:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788201686.012166:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94be00/0x9ae3109cee9e2686 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x105:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc4545 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201686.012178:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94a000/0x9ae3109cee9e267f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc4529 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:3.0:1788201686.012190:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788201686.012195:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e828953b480 x1875062187613312/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201686.012210:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187613312:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:3.0:1788201686.012298:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1000 going to sleep for 11 seconds 00000100:00100000:3.0:1788201686.014869:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b480 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187613312:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:3.0:1788201686.014915:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x105:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201686.014927:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201686.014952:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x105:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201686.014960:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x105:0x0](ffff8e8289e380c8) 00000080:00200000:3.0:1788201686.015059:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201686.015076:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201686.015127:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e268d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.015135:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e268d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.015142:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e268d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201686.015155:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187613440:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201686.015239:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201686.017344:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187613440:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201686.017370:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e268d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc456f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.017383:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e268d lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc456f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.017395:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e268d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc456f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.017406:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e268d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc456f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.017417:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e268d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc456f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.017425:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e268d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc456f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.017432:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e268d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc456f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201686.017444:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289539f80 x1875062187613440/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201702 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201686.017456:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e268d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc456f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.017595:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788201686.017604:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788201686.017637:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:3.0:1788201686.017643:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e268d 00000080:00010000:3.0:1788201686.017649:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201686.017651:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e268d from it ffffaa180174bc10 00010000:00010000:3.0:1788201686.017654:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e268d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc456f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.017663:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e268d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc456f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201686.017684:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:3.0:1788201686.017703:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-21, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:3.0:1788201686.017707:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-21, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:3.0:1788201686.017726:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-21,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788201686.017761:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2694 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.017768:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2694 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.017776:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2694 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788201686.017793:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187613568:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:3.0:1788201686.017900:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82911c1cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788201686.020701:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187613568:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:3.0:1788201686.020728:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2694 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc458b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.020749:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2694 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x106:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc458b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.020757:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2694 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc458b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.020768:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2694 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc458b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.020779:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2694 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc458b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.020792:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2694 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc458b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.020800:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2694 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc458b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.020808:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2694 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc458b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788201686.020820:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e8289539f80 x1875062187613568/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201702 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788201686.020832:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2694 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc458b expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788201686.020845:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2694 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc458b expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.020855:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/262 (0 0) 00000080:00200000:3.0:1788201686.020861:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788201686.020902:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x106:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788201686.020913:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788201686.020940:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x106:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788201686.021147:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x106:0x0](ffff8e8289e4a1c8): rc = 0 00000080:00200000:3.0:1788201686.021167:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x106:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:3.0:1788201686.021173:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x106:0x0](ffff8e8289e4a1c8) for lock 0x9ae3109cee9e2694 00000080:00010000:3.0:1788201686.021182:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x106:0x0](ffff8e8289e4a1c8) 00000080:00010000:3.0:1788201686.021186:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e2694 from it ffffaa180174be00 00010000:00010000:3.0:1788201686.021189:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2694 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc458b expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788201686.021198:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2694 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc458b expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201686.021226:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:3.0:1788201686.021242:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788201686.021251:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e268d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc456f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201686.021262:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e268d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc456f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788201686.021273:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788201686.021277:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:3.0:1788201686.021281:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e268d 00000080:00010000:3.0:1788201686.021285:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e268d from it ffffaa180174bd70 00010000:00010000:3.0:1788201686.021287:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e268d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc456f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788201686.021294:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e268d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc456f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788201686.021302:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:3.0:1788201686.021411:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-21, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201686.021664:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e268d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc456f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201686.021693:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2694 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc458b expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201686.021701:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x106:0x0]: losing layout lock 00010000:00010000:2.0:1788201686.021736:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2694 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc458b expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201686.021747:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e268d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc456f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201686.021753:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201686.021755:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94b800/0x9ae3109cee9e2694 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x106:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc458b expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201686.021772:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e828b94aa00/0x9ae3109cee9e268d lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc456f expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201686.021790:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201686.021798:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894e5500 x1875062187613824/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201686.021820:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187613824:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201686.021935:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201686.025916:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5500 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187613824:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201686.025974:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x106:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201686.025987:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201686.026028:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x106:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201686.026036:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x106:0x0](ffff8e8289e4a1c8) 00000080:00200000:2.0:1788201686.026123:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201686.026142:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201686.026196:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e269b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.026206:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e269b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.026216:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e269b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201686.026235:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187613952:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201686.026343:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201686.029378:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187613952:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201686.029416:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e269b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45b5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.029429:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e269b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45b5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.029440:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e269b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc45b5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.029451:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e269b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45b5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.029474:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e269b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc45b5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.029483:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e269b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc45b5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.029490:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e269b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc45b5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201686.029508:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894e5180 x1875062187613952/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201702 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201686.029522:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e269b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45b5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.029544:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201686.029552:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201686.029579:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201686.029585:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e269b 00000080:00010000:2.0:1788201686.029591:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201686.029594:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e269b from it ffffaa180174bc10 00010000:00010000:2.0:1788201686.029598:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e269b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc45b5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.029607:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e269b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45b5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201686.029637:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201686.029661:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-22, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201686.029665:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-22, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201686.029685:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-22,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201686.029724:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e26a2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.029731:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e26a2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.029744:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e26a2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201686.029765:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187614080:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201686.029838:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201686.035727:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187614080:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201686.035754:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e26a2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45d1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.035775:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e26a2 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x107:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45d1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.035784:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e26a2 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45d1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.035794:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e26a2 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc45d1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.035804:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e26a2 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45d1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.035817:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e26a2 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc45d1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.035825:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e26a2 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc45d1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.035840:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e26a2 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc45d1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201686.035851:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894e5180 x1875062187614080/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201702 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201686.035861:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e26a2 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45d1 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201686.035873:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e26a2 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45d1 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.035883:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/263 (0 0) 00000080:00200000:2.0:1788201686.035890:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201686.035932:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x107:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201686.035945:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201686.035972:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x107:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201686.036031:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x107:0x0](ffff8e8289c0d348): rc = 0 00000080:00200000:2.0:1788201686.036047:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x107:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201686.036052:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x107:0x0](ffff8e8289c0d348) for lock 0x9ae3109cee9e26a2 00000080:00010000:2.0:1788201686.036061:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x107:0x0](ffff8e8289c0d348) 00000080:00010000:2.0:1788201686.036065:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e26a2 from it ffffaa180174be00 00010000:00010000:2.0:1788201686.036068:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e26a2 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc45d1 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.036078:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e26a2 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc45d1 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201686.036114:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201686.036131:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201686.036140:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e269b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45b5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201686.036150:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e269b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45b5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201686.036156:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201686.036160:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201686.036163:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e269b 00000080:00010000:2.0:1788201686.036167:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e269b from it ffffaa180174bd70 00010000:00010000:2.0:1788201686.036170:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e269b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc45b5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201686.036177:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e269b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45b5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201686.036185:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201686.036192:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-22, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201686.036216:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e269b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc45b5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201686.036238:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e26a2 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc45d1 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201686.036248:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x107:0x0]: losing layout lock 00010000:00010000:2.0:1788201686.036278:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e26a2 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc45d1 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201686.036285:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e269b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc45b5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201686.036291:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201686.036294:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462800/0x9ae3109cee9e26a2 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x107:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc45d1 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201686.036337:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e269b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc45b5 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201686.036356:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201686.036364:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894e5180 x1875062187614336/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201686.036387:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187614336:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201686.036479:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201686.040908:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5180 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187614336:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201686.040969:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x107:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201686.040982:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201686.041022:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x107:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201686.041031:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x107:0x0](ffff8e8289c0d348) 00000080:00200000:2.0:1788201686.041121:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201686.041141:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201686.041197:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26a9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.041210:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26a9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.041220:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26a9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201686.041241:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187614464:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201686.041303:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201686.043817:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187614464:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201686.043841:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26a9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45fb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.043853:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26a9 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45fb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.043864:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26a9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc45fb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.043874:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26a9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45fb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.043885:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26a9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc45fb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.043893:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26a9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc45fb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.043906:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26a9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc45fb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201686.043918:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894e7b80 x1875062187614464/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201702 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201686.043932:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26a9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45fb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.043944:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201686.043950:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201686.043977:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201686.043983:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e26a9 00000080:00010000:2.0:1788201686.043988:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201686.043991:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e26a9 from it ffffaa180174bc10 00010000:00010000:2.0:1788201686.044140:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26a9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc45fb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.044152:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26a9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45fb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201686.044174:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201686.044194:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-23, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201686.044198:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-23, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201686.044219:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-23,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201686.044255:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26b0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.044263:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26b0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.044271:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26b0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201686.044289:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187614592:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201686.044548:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201686.047869:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187614592:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201686.047894:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26b0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4617 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.047916:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26b0 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x108:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4617 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.047925:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26b0 lrc: 4/1,0 mode: --/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4617 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.047935:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26b0 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4617 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.047945:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26b0 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4617 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.047958:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26b0 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4617 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.047972:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26b0 lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4617 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.047978:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26b0 lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4617 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201686.047989:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894e7b80 x1875062187614592/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201702 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201686.048220:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26b0 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4617 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201686.048234:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26b0 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4617 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.048244:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/264 (0 0) 00000080:00200000:2.0:1788201686.048251:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201686.048293:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x108:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201686.048304:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201686.048665:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x108:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201686.048722:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x108:0x0](ffff8e8292759148): rc = 0 00000080:00200000:2.0:1788201686.048737:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x108:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201686.048743:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x108:0x0](ffff8e8292759148) for lock 0x9ae3109cee9e26b0 00000080:00010000:2.0:1788201686.048753:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x108:0x0](ffff8e8292759148) 00000080:00010000:2.0:1788201686.048757:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e26b0 from it ffffaa180174be00 00010000:00010000:2.0:1788201686.048760:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26b0 lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc4617 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.048770:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26b0 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc4617 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201686.048801:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201686.048818:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201686.048829:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26a9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45fb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201686.048840:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26a9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45fb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201686.048847:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201686.048851:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201686.048854:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e26a9 00000080:00010000:2.0:1788201686.048858:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e26a9 from it ffffaa180174bd70 00010000:00010000:2.0:1788201686.048861:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26a9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc45fb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201686.048868:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26a9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc45fb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201686.048876:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201686.048883:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-23, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201686.048906:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26a9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc45fb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201686.048927:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26b0 lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc4617 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201686.048938:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x108:0x0]: losing layout lock 00010000:00010000:2.0:1788201686.048968:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26b0 lrc: 1/0,0 mode: --/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc4617 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201686.048976:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26a9 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc45fb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201686.048982:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201686.048985:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26b0 lrc: 0/0,0 mode: --/PR res: [0x200006991:0x108:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc4617 expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201686.049151:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26a9 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc45fb expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201686.049171:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201686.049181:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894e7b80 x1875062187614848/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201686.049202:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187614848:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201686.049369:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201686.052970:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7b80 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187614848:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201686.053178:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x108:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201686.053190:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201686.053212:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x108:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201686.053218:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x108:0x0](ffff8e8292759148) 00000080:00200000:2.0:1788201686.053299:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201686.053463:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201686.053506:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26b7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.053514:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26b7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.053520:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26b7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201686.053617:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187614976:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201686.053727:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201686.055345:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187614976:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201686.055363:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26b7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4641 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.055371:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26b7 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4641 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.055379:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26b7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4641 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.055386:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26b7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4641 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.055394:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26b7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4641 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.055399:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26b7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4641 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.055404:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26b7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4641 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201686.055412:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894e4700 x1875062187614976/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201702 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201686.055421:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26b7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4641 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.055429:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201686.055432:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201686.055452:0:42651:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201686.055457:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e26b7 00000080:00010000:2.0:1788201686.055462:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201686.055465:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e26b7 from it ffffaa180174bc10 00010000:00010000:2.0:1788201686.055468:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26b7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4641 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.055476:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26b7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4641 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201686.055497:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201686.055516:0:42651:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr-3-24, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=0 00000080:00200000:2.0:1788201686.055520:0:42651:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr-3-24, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=getattr 00000002:00010000:2.0:1788201686.055544:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr-3-24,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201686.055579:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26be lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.055586:0:42651:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26be lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.055594:0:42651:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26be lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42651 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201686.055672:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187615104:192.168.201.121@tcp:101:unlinkmany.0 00000100:00100000:2.0:1788201686.055725:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230e40 going to sleep for 11 seconds 00000100:00100000:2.0:1788201686.058080:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187615104:192.168.201.121@tcp:101:unlinkmany.0 00010000:00010000:2.0:1788201686.058107:0:42651:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26be lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc465d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.058126:0:42651:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26be lrc: 4/1,0 mode: --/PR res: [0x200006991:0x109:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc465d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.058134:0:42651:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26be lrc: 4/1,0 mode: --/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc465d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.058145:0:42651:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26be lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc465d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.058155:0:42651:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26be lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc465d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.058166:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26be lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc465d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.058178:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26be lrc: 5/2,0 mode: PR/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc465d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.058185:0:42651:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26be lrc: 4/1,0 mode: PR/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc465d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201686.058196:0:42651:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894e4700 x1875062187615104/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 592/640 e 0 to 0 dl 1788201702 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'unlinkmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201686.058206:0:42651:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26be lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc465d expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788201686.058218:0:42651:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26be lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc465d expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.058228:0:42651:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589961617/265 (0 0) 00000080:00200000:2.0:1788201686.058234:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201686.058273:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x109:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201686.058282:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201686.058305:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x109:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201686.058541:0:42651:0:(namei.c:147:ll_iget()) got inode: [0x200006991:0x109:0x0](ffff8e829275f448): rc = 0 00000080:00200000:2.0:1788201686.058557:0:42651:0:(file.c:6843:ll_layout_conf()) [0x200006991:0x109:0x0]: layout version change: 4294967294 -> 65535 00000080:00010000:2.0:1788201686.058562:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200006991:0x109:0x0](ffff8e829275f448) for lock 0x9ae3109cee9e26be 00000080:00010000:2.0:1788201686.058572:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200006991:0x109:0x0](ffff8e829275f448) 00000080:00010000:2.0:1788201686.058575:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e26be from it ffffaa180174be00 00010000:00010000:2.0:1788201686.058579:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26be lrc: 3/1,0 mode: PR/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x7cf2114380cc465d expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.058589:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26be lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x7cf2114380cc465d expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201686.058618:0:42651:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201686.058646:0:42651:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201686.058657:0:42651:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26b7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4641 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201686.058667:0:42651:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26b7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4641 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201686.058674:0:42651:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788201686.058678:0:42651:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201686.058680:0:42651:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e26b7 00000080:00010000:2.0:1788201686.058683:0:42651:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e26b7 from it ffffaa180174bd70 00010000:00010000:2.0:1788201686.058685:0:42651:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26b7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4641 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201686.058690:0:42651:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26b7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4641 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788201686.058695:0:42651:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 3 00000080:00200000:2.0:1788201686.058700:0:42651:0:(namei.c:2418:ll_unlink()) VFS Op:name=f12a.replay-vbr-3-24, dir=[0x200000007:0x1:0x0](ffff8e8292531988) 00010000:00010000:2.0:1788201686.058722:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26b7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc4641 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201686.058737:0:42651:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26be lrc: 2/0,0 mode: PR/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x7cf2114380cc465d expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201686.058743:0:42651:0:(vvp_object.c:117:vvp_conf_set()) [0x200006991:0x109:0x0]: losing layout lock 00010000:00010000:2.0:1788201686.058765:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26be lrc: 1/0,0 mode: --/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc465d expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201686.058770:0:42651:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26b7 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc4641 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788201686.058773:0:42651:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788201686.058775:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26be lrc: 0/0,0 mode: --/PR res: [0x200006991:0x109:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x7cf2114380cc465d expref: -99 pid: 42651 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201686.058785:0:42651:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26b7 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc4641 expref: -99 pid: 42651 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:2.0:1788201686.058796:0:42651:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8e8282a4b800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788201686.058802:0:42651:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8e82894e4700 x1875062187615360/t0(0) o36->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 504/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201686.058818:0:42651:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187615360:192.168.201.121@tcp:36:unlinkmany.0 00000100:00100000:2.0:1788201686.058915:0:42651:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e8292230480 going to sleep for 11 seconds 00000100:00100000:2.0:1788201686.063820:0:42651:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4700 pname:cluuid:pid:xid:nid:opc:job unlinkmany:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42651:1875062187615360:192.168.201.121@tcp:36:unlinkmany.0 00000080:00200000:2.0:1788201686.063881:0:42651:0:(vvp_io.c:1826:vvp_io_init()) [0x200006991:0x109:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788201686.063893:0:42651:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788201686.063929:0:42651:0:(vvp_io.c:304:vvp_io_fini()) [0x200006991:0x109:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788201686.063937:0:42651:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200006991:0x109:0x0](ffff8e829275f448) 00000080:00200000:2.0:1788201686.099169:0:42655:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988),name=/ 00000002:00010000:2.0:1788201686.099194:0:42655:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788201686.099246:0:42655:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26c5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 42655 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.099253:0:42655:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26c5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42655 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.099261:0:42655:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26c5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 42655 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201686.099280:0:42655:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job checkstat:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42655:1875062187615488:192.168.201.121@tcp:101:checkstat.0 00000100:00100000:2.0:1788201686.099384:0:42655:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82922300c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201686.102559:0:42655:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job checkstat:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42655:1875062187615488:192.168.201.121@tcp:101:checkstat.0 00010000:00010000:2.0:1788201686.102587:0:42655:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26c5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4687 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.102611:0:42655:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26c5 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4687 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.102624:0:42655:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26c5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4687 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.102647:0:42655:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26c5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4687 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.102661:0:42655:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26c5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4687 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.102670:0:42655:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26c5 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4687 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.102677:0:42655:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26c5 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4687 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201686.102690:0:42655:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8e82894e4380 x1875062187615488/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 576/688 e 0 to 0 dl 1788201702 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201686.102705:0:42655:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26c5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4687 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.102718:0:42655:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788201686.102725:0:42655:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788201686.102755:0:42655:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 20480 00000080:00010000:2.0:1788201686.102762:0:42655:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) for lock 0x9ae3109cee9e26c5 00000080:00010000:2.0:1788201686.102768:0:42655:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8e8292531988) 00000080:00010000:2.0:1788201686.102772:0:42655:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e26c5 from it ffffaa180198bb58 00010000:00010000:2.0:1788201686.102775:0:42655:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26c5 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc4687 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.102785:0:42655:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26c5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc4687 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201686.102812:0:42655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8e8292531988), inode mode 41ed mask 1 00000080:00200000:2.0:1788201686.102831:0:42655:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d12a.replay-vbr, dir=[0x200000007:0x1:0x0](ffff8e8292531988), flags=24 00000080:00200000:2.0:1788201686.102836:0:42655:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d12a.replay-vbr, dir=[0x200000007:0x1:0x0](ffff8e8292531988), intent=0 00000002:00010000:2.0:1788201686.102857:0:42655:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d12a.replay-vbr,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1788201686.102903:0:42655:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42655 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.102911:0:42655:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42655 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.102935:0:42655:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42655 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201686.102956:0:42655:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job checkstat:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42655:1875062187615616:192.168.201.121@tcp:101:checkstat.0 00000100:00100000:2.0:1788201686.103024:0:42655:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82922300c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201686.105398:0:42655:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job checkstat:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42655:1875062187615616:192.168.201.121@tcp:101:checkstat.0 00010000:00010000:2.0:1788201686.105425:0:42655:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc46a3 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.105443:0:42655:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 4/1,0 mode: --/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc46a3 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.105451:0:42655:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 4/1,0 mode: --/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc46a3 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.105461:0:42655:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 4/1,0 mode: PR/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc46a3 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.105471:0:42655:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 4/1,0 mode: PR/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc46a3 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.105483:0:42655:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 5/2,0 mode: PR/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc46a3 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.105498:0:42655:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 5/2,0 mode: PR/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc46a3 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.105506:0:42655:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 4/1,0 mode: PR/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc46a3 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201686.105518:0:42655:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff8e82894e4380 x1875062187615616/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 584/688 e 0 to 0 dl 1788201702 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'checkstat.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788201686.105532:0:42655:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc46a3 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.105541:0:42655:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8e8282af5400 type 13 mode 4 res 8589971617/1 (0 0) 00000080:00200000:2.0:1788201686.105548:0:42655:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788201686.105574:0:42655:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x2000090a1:0x1:0x0], updating i_size 4096 00000080:00200000:2.0:1788201686.105577:0:42655:0:(namei.c:147:ll_iget()) got inode: [0x2000090a1:0x1:0x0](ffff8e8289e40908): rc = 0 00000080:00010000:2.0:1788201686.105583:0:42655:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x2000090a1:0x1:0x0](ffff8e8289e40908) for lock 0x9ae3109cee9e26cc 00000080:00010000:2.0:1788201686.105595:0:42655:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x2000090a1:0x1:0x0](ffff8e8289e40908) 00000080:00010000:2.0:1788201686.105599:0:42655:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e26cc from it ffffaa180198ba28 00010000:00010000:2.0:1788201686.105613:0:42655:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc46a3 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.105623:0:42655:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc46a3 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788201686.105664:0:42655:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x2000090a1:0x1:0x0](ffff8e8289e40908), inode mode 41ed mask 1 00000080:00200000:2.0:1788201686.105682:0:42655:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f12a.replay-vbr, dir=[0x2000090a1:0x1:0x0](ffff8e8289e40908), flags=8 00000080:00200000:2.0:1788201686.105686:0:42655:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f12a.replay-vbr, dir=[0x2000090a1:0x1:0x0](ffff8e8289e40908), intent=getattr 00000002:00010000:2.0:1788201686.105706:0:42655:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f12a.replay-vbr,[0x0:0x0:0x0]) in obj [0x2000090a1:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788201686.105745:0:42655:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e26d3 lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 42655 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.105752:0:42655:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e26d3 lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42655 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.105761:0:42655:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e26d3 lrc: 3/1,0 mode: --/CR res: [0x2000090a1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42655 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788201686.105779:0:42655:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job checkstat:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42655:1875062187615744:192.168.201.121@tcp:101:checkstat.0 00000100:00100000:2.0:1788201686.105842:0:42655:0:(client.c:2715:ptlrpc_set_wait()) set ffff8e82922300c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788201686.107817:0:42655:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job checkstat:dc31b02e-cf70-4b83-9f70-d7124af83e4d:42655:1875062187615744:192.168.201.121@tcp:101:checkstat.0 00010000:00010000:2.0:1788201686.107836:0:42655:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e26d3 lrc: 4/1,0 mode: --/CR res: [0x2000090a1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 42655 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.107848:0:42655:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e26d3 lrc: 4/1,0 mode: --/CR res: [0x2000090a1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 42655 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.107854:0:42655:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e26d3 lrc: 4/1,0 mode: --/CR res: [0x2000090a1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 42655 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.107860:0:42655:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e26d3 lrc: 3/0,0 mode: --/CR res: [0x2000090a1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 42655 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.107868:0:42655:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e26d3 lrc: 4/0,0 mode: --/CR res: [0x2000090a1:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 42655 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.107874:0:42655:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8e8289463800 already unused, calling callback (ffffffffc1c51460) 00010000:00010000:2.0:1788201686.107878:0:42655:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e26d3 lrc: 4/0,0 mode: --/CR res: [0x2000090a1:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 42655 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.107885:0:42655:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e26d3 lrc: 4/0,0 mode: --/CR res: [0x2000090a1:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 42655 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.107892:0:42655:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e26d3 lrc: 5/0,0 mode: --/CR res: [0x2000090a1:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 42655 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.107899:0:42655:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788201686.107904:0:42655:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e26d3 lrc: 3/0,0 mode: --/CR res: [0x2000090a1:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 42655 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788201686.107910:0:42655:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463800/0x9ae3109cee9e26d3 lrc: 0/0,0 mode: --/CR res: [0x2000090a1:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 42655 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788201686.107942:0:42655:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff8e82894e4380 x1875062187615744/t0(0) o101->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 584/584 e 0 to 0 dl 1788201702 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'checkstat.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788201686.107957:0:42655:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x2000090a1:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1788201686.107979:0:42655:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc46a3 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201686.107991:0:42655:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 2/1,0 mode: PR/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc46a3 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1788201686.107999:0:42655:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x9ae3109cee9e26cc from it ffffaa180198b930 00010000:00010000:2.0:1788201686.108003:0:42655:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 3/1,0 mode: PR/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x7cf2114380cc46a3 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788201686.108011:0:42655:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x7cf2114380cc46a3 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788201686.212477:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201687.242071:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788201688.260562:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201689.285634:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201690.308808:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201690.633521:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2404:1875062187615872:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201690.633634:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187616128:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201690.633657:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187616384:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201690.633682:0:2404:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788201690.633687:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187616000:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201690.633707:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187616256:192.168.201.121@tcp:400:kworker.0 00010000:00010000:3.0:1788201691.332768:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201692.359124:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201693.380474:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201694.404679:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201695.428567:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788201695.748599:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187616640:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201695.748600:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187616512:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201695.748762:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187616896:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201695.748763:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187616768:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201695.748837:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187617024:192.168.201.121@tcp:400:kworker.0 00010000:00010000:1.0:1788201696.453784:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201697.476549:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201698.500554:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201699.524509:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201700.548515:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201700.868623:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187617280:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201700.868631:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2403:1875062187617152:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201700.868765:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187617408:192.168.201.121@tcp:400:kworker.0 00000100:00100000:3.0:1788201700.868766:0:2404:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187617536:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201700.868842:0:2403:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187617664:192.168.201.121@tcp:400:kworker.0 00010000:00010000:0.0:1788201701.572528:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201702.596550:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201703.620441:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201704.644484:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201705.668463:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788201705.988504:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187617792:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201705.988507:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187617920:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201705.988603:0:2401:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187618176:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201705.988624:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187618048:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201705.988661:0:2402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187618304:192.168.201.121@tcp:400:kworker.0 00010000:00010000:0.0:1788201706.692475:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788201707.012388:0:2404:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201690/real 1788201690] req@ffff8e8289539500 x1875062187616000/t0(0) o400->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201706 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201707.012412:0:2404:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8e8289539500 x1875062187616000/t0(0) o400->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201706 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201707.012420:0:2404:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1788201707.012422:0:2404:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8e8282a4b800: Connection to lustre-MDT0000 (at 192.168.201.121@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1788201707.012427:0:2404:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00000400:2.0:1788201707.012445:0:2404:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201690/real 1788201690] req@ffff8e8289539880 x1875062187616256/t0(0) o400->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201706 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201707.012457:0:2404:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8e8289539880 x1875062187616256/t0(0) o400->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201706 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201707.012463:0:2404:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1788201707.012467:0:2404:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff8e8282a4b800: Connection to lustre-OST0000 (at 192.168.201.121@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1788201707.012471:0:2404:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0000_UUID: waking up pinger s:DISCONN 00000100:00000400:2.0:1788201707.012486:0:2404:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201690/real 1788201690] req@ffff8e8289539f80 x1875062187615872/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201706 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201707.012502:0:2404:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8e8289539f80 x1875062187615872/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201706 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201707.012508:0:2404:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from FULL to DISCONN 00000100:02020000:2.0:1788201707.012510:0:2404:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.201.121@tcp: Connection to MGS (at 192.168.201.121@tcp) was lost; in progress operations using this service will fail 00000100:00080000:2.0:1788201707.012513:0:2404:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.201.121@tcp for MGC192.168.201.121@tcp not replayable, auto-deactivating 00000100:00080000:2.0:1788201707.012515:0:2404:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:2.0:1788201707.012519:0:2404:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8e8289539f80 x1875062187615872/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 1 dl 1788201706 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201707.012527:0:2404:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8e828793c000 x1875062187616512/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201711 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201707.012544:0:2404:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8e82894e4380 x1875062187617152/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201716 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201707.012558:0:2404:0:(client.c:3589:ptlrpc_abort_inflight()) @@@ inflight req@ffff8e828793f100 x1875062187617792/t0(0) o400->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 224/224 e 0 to 0 dl 1788201721 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:2.0:1788201707.012574:0:2404:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:2.0:1788201707.012577:0:2404:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:2.0:1788201707.012579:0:2404:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:2.0:1788201707.012581:0:2404:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00080000:0.0:1788201707.012583:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788201707.012585:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2401:1875062187616512:192.168.201.121@tcp:400:kworker.0 00000100:00080000:0.0:1788201707.012587:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201707.012593:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000400:2.0:1788201707.012596:0:2404:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201690/real 1788201690] req@ffff8e828953a680 x1875062187616128/t0(0) o400->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201706 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201707.012599:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3074 00000100:00100000:3.0:1788201707.012606:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2403:1875062187617152:192.168.201.121@tcp:400:kworker.0 00000100:00080000:0.0:1788201707.012613:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788201707.012613:0:2404:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8e828953a680 x1875062187616128/t0(0) o400->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201706 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201707.012616:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:0:1875062187618432:192.168.201.121@tcp:400:kworker.0 00000008:00100000:0.0:1788201707.012623:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1788201707.012628:0:2404:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1788201707.012632:0:2404:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0001-mdc-ffff8e8282a4b800: Connection to lustre-MDT0001 (at 192.168.201.121@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1788201707.012637:0:2404:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0001_UUID: waking up pinger s:DISCONN 00000100:00100000:0.0:1788201707.012640:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793df80 x1875062187618560/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000400:2.0:1788201707.012646:0:2404:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201690/real 1788201690] req@ffff8e8289538380 x1875062187616384/t0(0) o400->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201706 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201707.012653:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2402:1875062187617792:192.168.201.121@tcp:400:kworker.0 00000100:00080000:0.0:1788201707.012655:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201707.012660:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201707.012663:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788201707.012663:0:2404:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8e8289538380 x1875062187616384/t0(0) o400->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201706 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201707.012667:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 298 00000100:00080000:2.0:1788201707.012670:0:2404:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from FULL to DISCONN 00000100:00080000:0.0:1788201707.012673:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:02000400:2.0:1788201707.012674:0:2404:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0001-osc-ffff8e8282a4b800: Connection to lustre-OST0001 (at 192.168.201.121@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1788201707.012676:0:2404:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0001_UUID: waking up pinger s:DISCONN 00000008:00100000:0.0:1788201707.012677:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201707.012680:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187618560:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201707.012689:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187618688/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201707.012694:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187616000:192.168.201.121@tcp:400:kworker.0 00000100:00080000:0.0:1788201707.012700:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788201707.012700:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187616256:192.168.201.121@tcp:400:kworker.0 00000100:00080000:0.0:1788201707.012703:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201707.012705:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201707.012708:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 1356 00000100:00100000:2.0:1788201707.012710:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2404:1875062187615872:192.168.201.121@tcp:400:kworker.0 00000100:00080000:0.0:1788201707.012713:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788201707.012715:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187616128:192.168.201.121@tcp:400:kworker.0 00000008:00100000:0.0:1788201707.012717:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201707.012726:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187618816/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201707.012729:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187616384:192.168.201.121@tcp:400:kworker.0 00000100:00080000:0.0:1788201707.012735:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201707.012738:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201707.012740:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201707.012743:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 1332 00000100:00080000:0.0:1788201707.012748:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201707.012753:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788201707.012763:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187618688:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201707.012766:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187618944/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201707.012783:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201707.012787:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201707.012789:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201707.012792:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3072 00000100:00100000:3.0:1788201707.012795:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187618816:192.168.201.121@tcp:8:kworker.0 00000100:00080000:0.0:1788201707.012800:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201707.012813:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187619072/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201707.012842:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187618944:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201707.012873:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187619072:192.168.201.121@tcp:250:kworker.0 00010000:00010000:0.0:1788201707.716416:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201708.740474:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201709.764451:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201710.788432:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:1.0:1788201711.108415:0:2402:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201695/real 1788201695] req@ffff8e828793f800 x1875062187616640/t0(0) o400->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201711 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:0.0:1788201711.108416:0:2401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201695/real 1788201695] req@ffff8e828793d500 x1875062187616768/t0(0) o400->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201711 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201711.108467:0:2402:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8e828793f800 x1875062187616640/t0(0) o400->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201711 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201711.108469:0:2401:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8e828793d500 x1875062187616768/t0(0) o400->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201711 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201711.108495:0:2401:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8e8291097000 already not connected (conn 2, was 1): CONNECTING 00000100:00080000:1.0:1788201711.108496:0:2402:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8e82871d2800 already not connected (conn 154, was 153): CONNECTING 00000100:00000400:0.0:1788201711.108503:0:2401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201695/real 1788201695] req@ffff8e828793dc00 x1875062187617024/t0(0) o400->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201711 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788201711.108507:0:2402:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201695/real 1788201695] req@ffff8e828793e680 x1875062187616896/t0(0) o400->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201711 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201711.108521:0:2401:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8e828793dc00 x1875062187617024/t0(0) o400->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201711 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201711.108528:0:2402:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8e828793e680 x1875062187616896/t0(0) o400->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201711 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201711.108529:0:2401:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8e8291090000 already not connected (conn 12, was 11): CONNECTING 00000100:00080000:1.0:1788201711.108538:0:2402:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8e8291096800 already not connected (conn 6, was 5): CONNECTING 00000100:00100000:0.0:1788201711.108549:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187616768:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201711.108549:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187616640:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201711.108565:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187617024:192.168.201.121@tcp:400:kworker.0 00000100:00100000:1.0:1788201711.108566:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187616896:192.168.201.121@tcp:400:kworker.0 00010000:00010000:1.0:1788201711.812435:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201712.132388:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201712.132398:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201712.132402:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201712.132406:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201712.132409:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201712.132413:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201712.132415:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201712.132418:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201712.132421:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201712.132425:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1788201712.836450:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201713.860476:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201714.884420:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201715.908462:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788201716.164422:0:2404:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201700/real 1788201700] req@ffff8e82894e5c00 x1875062187617280/t0(0) o400->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201716 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201716.164448:0:2404:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8e82894e5c00 x1875062187617280/t0(0) o400->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201716 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201716.164454:0:2404:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8e82871d2800 already not connected (conn 154, was 153): CONNECTING 00000100:00000400:2.0:1788201716.164459:0:2404:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201700/real 1788201700] req@ffff8e82894e5f80 x1875062187617536/t0(0) o400->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201716 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201716.164470:0:2404:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8e82894e5f80 x1875062187617536/t0(0) o400->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201716 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201716.164474:0:2404:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8e8291096800 already not connected (conn 6, was 5): CONNECTING 00000100:00100000:2.0:1788201716.164485:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187617280:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201716.164489:0:2404:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2404:1875062187617536:192.168.201.121@tcp:400:kworker.0 00000100:00000400:2.0:1788201716.228405:0:2403:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201700/real 1788201700] req@ffff8e82894e6d80 x1875062187617408/t0(0) o400->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201716 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201716.228437:0:2403:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8e82894e6d80 x1875062187617408/t0(0) o400->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201716 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201716.228443:0:2403:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8e8291097000 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:2.0:1788201716.228448:0:2403:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201700/real 1788201700] req@ffff8e82894e4a80 x1875062187617664/t0(0) o400->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201716 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201716.228456:0:2403:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8e82894e4a80 x1875062187617664/t0(0) o400->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201716 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201716.228461:0:2403:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8e8291090000 already not connected (conn 12, was 11): CONNECTING 00000100:00100000:2.0:1788201716.228468:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187617408:192.168.201.121@tcp:400:kworker.0 00000100:00100000:2.0:1788201716.228472:0:2403:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2403:1875062187617664:192.168.201.121@tcp:400:kworker.0 00010000:00010000:0.0:1788201716.932464:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201717.252430:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201717.252444:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201717.252450:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788201717.252450:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201707/real 1788201707] req@ffff8e828793e300 x1875062187618944/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201717 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201717.252457:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201717.252465:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201717.252471:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201717.252476:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201717.252479:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201717.252483:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201717.252496:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:3.0:1788201717.252498:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187618944/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201717 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201717.252579:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201707/real 1788201707] req@ffff8e828793f100 x1875062187619072/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201717 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201717.252594:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187619072/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201717 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201717.252605:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201707/real 1788201707] req@ffff8e828793c700 x1875062187618688/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201717 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201717.252617:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187618688/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201717 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201717.252626:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201707/real 1788201707] req@ffff8e828793ca80 x1875062187618816/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201717 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201717.252638:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187618816/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201717 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201717.252648:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201707/real 1788201707] req@ffff8e828793df80 x1875062187618560/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201717 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201717.252660:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793df80 x1875062187618560/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201717 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201717.252678:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201717.252697:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201717.252701:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187618944:192.168.201.121@tcp:8:kworker.0 00000100:00080000:3.0:1788201717.252708:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201717.252714:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201717.252718:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187619072:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788201717.252725:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201717.252726:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201717.252732:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201717.252735:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201717.252741:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00080000:0.0:1788201717.252744:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3167 00000100:00100000:3.0:1788201717.252750:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187618688:192.168.201.121@tcp:38:kworker.0 00000100:00080000:3.0:1788201717.252759:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201717.252761:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788201717.252768:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00080000:0.0:1788201717.252774:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788201717.252777:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187618816:192.168.201.121@tcp:8:kworker.0 00000100:00080000:3.0:1788201717.252785:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201717.252799:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201717.252803:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187618560:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201717.252817:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187619200/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201717.252836:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201717.252840:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201717.252842:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201717.252846:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3167 00000100:00080000:0.0:1788201717.252850:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1788201717.252853:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788201717.252858:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187619200:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788201717.252864:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201717.252878:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187619328/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201717.252889:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201717.252892:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201717.252894:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201717.252897:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3167 00000100:00080000:0.0:1788201717.252901:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1788201717.252904:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201717.252909:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201717.252920:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187619456/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201717.252929:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201717.252932:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201717.252934:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201717.252937:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3167 00000100:00080000:0.0:1788201717.252940:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1788201717.252944:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201717.252950:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201717.252963:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187619584/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201717.252973:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201717.252976:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201717.252978:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201717.252982:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3167 00000100:00080000:0.0:1788201717.252990:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 5s 00000100:00100000:3.0:1788201717.252990:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187619328:192.168.201.121@tcp:38:kworker.0 00000100:00080000:0.0:1788201717.252999:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201717.253006:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201717.253024:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793df80 x1875062187619712/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201717.253055:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187619456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201717.253112:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187619584:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201717.253157:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187619712:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201717.956508:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201718.980480:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201720.004461:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201721.028599:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1788201721.348435:0:2402:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201705/real 1788201705] req@ffff8e828793f480 x1875062187618048/t0(0) o400->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201721 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788201721.348435:0:2401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201705/real 1788201705] req@ffff8e828793fb80 x1875062187617920/t0(0) o400->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201721 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201721.348473:0:2401:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8e828793fb80 x1875062187617920/t0(0) o400->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201721 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201721.348482:0:2402:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8e828793f480 x1875062187618048/t0(0) o400->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 224/224 e 0 to 1 dl 1788201721 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201721.348482:0:2401:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8e82871d2800 already not connected (conn 155, was 153): CONNECTING 00000100:00000400:1.0:1788201721.348492:0:2401:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201705/real 1788201705] req@ffff8e828793c380 x1875062187618176/t0(0) o400->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201721 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201721.348497:0:2402:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8e8291097000 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:0.0:1788201721.348507:0:2402:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201705/real 1788201705] req@ffff8e828793ce00 x1875062187618304/t0(0) o400->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201721 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201721.348515:0:2401:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8e828793c380 x1875062187618176/t0(0) o400->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201721 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201721.348522:0:2401:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8e8291096800 already not connected (conn 7, was 5): CONNECTING 00000100:00100000:0.0:1788201721.348529:0:2402:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8e828793ce00 x1875062187618304/t0(0) o400->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 224/224 e 0 to 1 dl 1788201721 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201721.348537:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187617920:192.168.201.121@tcp:400:kworker.0 00000100:00080000:0.0:1788201721.348542:0:2402:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8e8291090000 already not connected (conn 13, was 11): CONNECTING 00000100:00100000:1.0:1788201721.348545:0:2401:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2401:1875062187618176:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201721.348556:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187618048:192.168.201.121@tcp:400:kworker.0 00000100:00100000:0.0:1788201721.348566:0:2402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2402:1875062187618304:192.168.201.121@tcp:400:kworker.0 00010000:00010000:1.0:1788201722.052466:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201722.372400:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201722.372408:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201722.372411:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201722.372415:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201722.372417:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201722.372420:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201722.372423:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201722.372425:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201722.372428:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201722.372432:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1788201723.076474:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201724.100500:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201725.124489:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201726.148502:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201727.172494:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201727.492405:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201727.492415:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201727.492418:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201727.492421:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201727.492423:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201727.492425:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201727.492426:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201727.492428:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788201727.492430:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201727.492431:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1788201728.196464:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201729.220432:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201730.244496:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201731.268493:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201732.292528:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201732.612437:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788201732.612445:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201717/real 1788201717] req@ffff8e828793df80 x1875062187619712/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201732 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201732.612447:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788201732.612451:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201732.612458:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788201732.612462:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201732.612466:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788201732.612468:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201732.612471:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788201732.612472:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201732.612474:0:37:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788201732.612489:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793df80 x1875062187619712/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201732 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201732.612498:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201717/real 1788201717] req@ffff8e828793d500 x1875062187619328/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201732 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201732.612506:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187619328/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201732 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201732.612512:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201717/real 1788201717] req@ffff8e828793d880 x1875062187619456/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201732 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201732.612518:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187619456/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201732 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201732.612524:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201717/real 1788201717] req@ffff8e828793c000 x1875062187619584/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201732 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201732.612531:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187619584/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201732 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201732.612536:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201717/real 1788201717] req@ffff8e828793dc00 x1875062187619200/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201732 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201732.612542:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187619200/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201732 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201732.612561:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201732.612578:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201732.612582:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187619712:192.168.201.121@tcp:8:kworker.0 00000100:00080000:0.0:1788201732.612587:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201732.612592:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201732.612596:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187619328:192.168.201.121@tcp:38:kworker.0 00000100:00080000:0.0:1788201732.612601:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201732.612604:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201732.612610:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201732.612611:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201732.612617:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187619456:192.168.201.121@tcp:38:kworker.0 00000100:00080000:2.0:1788201732.612617:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201732.612625:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201732.612626:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201732.612629:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201732.612633:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201732.612636:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187619584:192.168.201.121@tcp:8:kworker.0 00000100:00080000:2.0:1788201732.612639:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3177 00000100:00080000:0.0:1788201732.612643:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201732.612650:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:2.0:1788201732.612651:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 10s 00000100:00100000:0.0:1788201732.612654:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187619200:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788201732.612661:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201732.612673:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201732.612703:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187619840/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201732.612727:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201732.612730:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201732.612732:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201732.612737:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3177 00000100:00080000:2.0:1788201732.612741:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1788201732.612744:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201732.612749:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201732.612755:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187619840:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201732.612760:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187619968/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201732.612768:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201732.612770:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788201732.612771:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201732.612773:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3177 00000100:00080000:2.0:1788201732.612776:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1788201732.612778:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201732.612781:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201732.612789:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187620096/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201732.612803:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201732.612806:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788201732.612807:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201732.612811:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3177 00000100:00080000:2.0:1788201732.612815:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1788201732.612830:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201732.612836:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201732.612849:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187620224/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201732.612859:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201732.612862:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201732.612863:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201732.612866:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3177 00000100:00080000:2.0:1788201732.612870:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1788201732.612873:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201732.612881:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187619968:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201732.612884:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187620352/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201732.612907:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187620096:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788201732.612927:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187620224:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788201732.612948:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187620352:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788201733.316512:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201734.340536:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201735.364504:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201736.388458:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201737.412552:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201737.732488:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201737.732499:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201737.732503:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201737.732507:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201737.732510:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201737.732513:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201737.732516:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201737.732519:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201737.732522:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201737.732524:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:0.0:1788201738.436629:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201739.460479:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201740.484496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201741.508473:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201742.532504:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201742.852435:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201742.852446:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201742.852449:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201742.852453:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201742.852456:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201742.852460:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201742.852463:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201742.852466:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788201742.852469:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201742.852472:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000800:00000100:2.0:1788201743.492403:0:2392:0:(socklnd_cb.c:2254:ksocknal_find_timed_out_conn()) A connection with 12345-192.168.201.121@tcp (192.168.201.121:988) timed out; the network or node may be down. 00010000:00010000:1.0:1788201743.556485:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201744.580443:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201745.604488:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201746.628463:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201747.652477:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201747.972427:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201747.972439:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201747.972443:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201747.972447:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201747.972450:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201747.972454:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201747.972457:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201747.972461:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788201747.972464:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201747.972468:0:11298:0:(pinger.c:247:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1788201748.676491:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201749.700461:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201750.724499:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201751.748554:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788201752.068431:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201732/real 1788201732] req@ffff8e82894e7100 x1875062187620352/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201752.068464:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187620352/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201752 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201752.068477:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201732/real 1788201732] req@ffff8e82894e6d80 x1875062187619968/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201752.068507:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187619968/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201752 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201752.068519:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201732/real 1788201732] req@ffff8e82894e5f80 x1875062187620096/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201752.068532:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187620096/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201752 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201752.068542:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201732/real 1788201732] req@ffff8e82894e5c00 x1875062187620224/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201752.068554:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187620224/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201752 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201752.068564:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788201732/real 1788201732] req@ffff8e82894e4a80 x1875062187619840/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201752.068576:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187619840/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201752 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201752.068597:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201752.068620:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201752.068625:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187620352:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788201752.068631:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201752.068640:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201752.068645:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187619968:192.168.201.121@tcp:38:kworker.0 00000100:00080000:0.0:1788201752.068652:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201752.068652:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201752.068659:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201752.068662:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201752.068668:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3192 00000100:00080000:3.0:1788201752.068668:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201752.068677:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187620096:192.168.201.121@tcp:8:kworker.0 00000100:00080000:0.0:1788201752.068680:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788201752.068687:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201752.068689:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:3.0:1788201752.068704:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201752.068711:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187620224:192.168.201.121@tcp:8:kworker.0 00000100:00080000:3.0:1788201752.068719:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788201752.068724:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187620480/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201752.068736:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00080000:0.0:1788201752.068738:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201752.068743:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201752.068746:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788201752.068747:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187619840:192.168.201.121@tcp:38:kworker.0 00000100:00080000:0.0:1788201752.068753:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3192 00000100:00080000:0.0:1788201752.068757:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1788201752.068759:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201752.068766:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201752.068775:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187620608/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201752.068783:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201752.068786:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201752.068787:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201752.068790:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3192 00000100:00080000:0.0:1788201752.068792:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1788201752.068798:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788201752.068799:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187620480:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788201752.068801:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201752.068811:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187620736/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201752.068818:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201752.068820:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201752.068821:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201752.068823:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3192 00000100:00080000:0.0:1788201752.068826:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1788201752.068828:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201752.068831:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201752.068840:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187620864/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201752.068845:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201752.068847:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201752.068848:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201752.068850:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3192 00000100:00080000:0.0:1788201752.068853:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 15s 00000100:00080000:0.0:1788201752.068855:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201752.068858:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201752.068865:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793df80 x1875062187620992/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201752.068904:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187620608:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201752.068942:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187620736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201752.068970:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187620864:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201752.069004:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187620992:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201752.772490:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788201752.900606:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.201.121:988 00000400:00000100:1.0:1788201752.900656:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788201752.900672:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788201752.900707:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788201752.900716:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788201752.900726:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788201752.900736:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:3.0:1788201752.900758:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201752/real 1788201752] req@ffff8e828793d880 x1875062187620736/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201752.900788:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187620736/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201752.900811:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201752.900816:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201752.900819:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187620736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201752.900825:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201752/real 1788201752] req@ffff8e828793d500 x1875062187620864/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201752.900834:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187620864/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201752.900840:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201752.900842:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201752.900844:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187620864:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201752.900848:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201752/real 1788201752] req@ffff8e828793df80 x1875062187620992/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201752.900855:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793df80 x1875062187620992/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201752.900860:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201752.900862:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201752.900863:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187620992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201752.900867:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201752/real 1788201752] req@ffff8e828793dc00 x1875062187620480/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201752.900874:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187620480/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201752.900880:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201752.900882:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201752.900884:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187620480:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788201752.900888:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201752/real 1788201752] req@ffff8e828793c000 x1875062187620608/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201752.900894:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187620608/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201752.900899:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201752.900901:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201752.900903:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187620608:192.168.201.121@tcp:38:kworker.0 00010000:00010000:0.0:1788201753.796473:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201754.820489:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201755.844466:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201756.868437:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201757.188426:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201757.188435:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201757.188439:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201757.188448:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3212 00000100:00080000:2.0:1788201757.188457:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201757.188463:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788201757.188495:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187621120/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201757.188516:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201757.188519:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201757.188520:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201757.188523:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3212 00000100:00080000:2.0:1788201757.188525:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201757.188530:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201757.188537:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201757.188546:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187621248/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201757.188555:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788201757.188555:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187621120:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788201757.188562:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201757.188565:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201757.188568:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3212 00000100:00080000:2.0:1788201757.188569:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201757.188572:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201757.188574:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201757.188583:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187621376/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201757.188589:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201757.188591:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788201757.188592:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201757.188595:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3212 00000100:00080000:2.0:1788201757.188611:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201757.188614:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201757.188617:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201757.188627:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187621504/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201757.188636:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201757.188640:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788201757.188642:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201757.188646:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3212 00000100:00080000:2.0:1788201757.188651:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201757.188656:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201757.188663:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788201757.188668:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187621248:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201757.188679:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187621632/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201757.188704:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187621376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201757.188725:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187621504:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201757.188748:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187621632:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201757.892513:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201758.916453:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788201759.364670:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.201.121:988 00000400:00000100:0.0:1788201759.364727:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788201759.364747:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:0.0:1788201759.364791:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:0.0:1788201759.364803:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:0.0:1788201759.364814:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:0.0:1788201759.364826:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:3.0:1788201759.364841:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201757/real 1788201759] req@ffff8e82894e6300 x1875062187621248/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201759.364876:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187621248/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201759.364896:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201759.364902:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201759.364907:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187621248:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201759.364917:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201757/real 1788201759] req@ffff8e82894e7800 x1875062187621376/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201759.364938:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187621376/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201759.364948:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201759.364951:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201759.364954:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187621376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201759.364960:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201757/real 1788201759] req@ffff8e82894e4e00 x1875062187621504/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201759.364969:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187621504/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201759.364975:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201759.364978:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201759.364980:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187621504:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201759.364984:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201757/real 1788201759] req@ffff8e82894e7480 x1875062187621120/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201759.364995:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187621120/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201759.365004:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201759.365008:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201759.365013:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187621120:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788201759.365047:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201757/real 1788201759] req@ffff8e82894e6680 x1875062187621632/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201759.365060:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187621632/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201759.365069:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201759.365072:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201759.365076:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187621632:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201759.940522:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201760.964447:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201761.988486:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201762.308430:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201762.308440:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201762.308445:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201762.308457:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3217 00000100:00080000:1.0:1788201762.308468:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201762.308475:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788201762.308516:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187621760/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201762.308547:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201762.308550:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201762.308551:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201762.308555:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3217 00000100:00080000:1.0:1788201762.308558:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201762.308561:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201762.308571:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201762.308589:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187621888/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201762.308600:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201762.308604:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788201762.308607:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201762.308611:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3217 00000100:00080000:1.0:1788201762.308616:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201762.308621:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201762.308628:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201762.308641:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187621760:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788201762.308648:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187622016/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201762.308665:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201762.308671:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201762.308673:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201762.308679:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3217 00000100:00080000:1.0:1788201762.308683:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201762.308690:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201762.308699:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788201762.308715:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187622144/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201762.308727:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201762.308731:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201762.308734:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201762.308740:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3217 00000100:00080000:1.0:1788201762.308745:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201762.308751:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201762.308758:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788201762.308768:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187621888:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788201762.308778:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187622272/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201762.308834:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187622016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201762.308868:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187622144:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201762.308898:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187622272:192.168.201.121@tcp:8:kworker.0 00000400:00000100:2.0:1788201762.436496:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.201.121:988 00000400:00000100:2.0:1788201762.436541:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788201762.436559:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788201762.436598:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788201762.436608:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788201762.436620:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788201762.436631:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:3.0:1788201762.436638:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201762/real 1788201762] req@ffff8e828d722680 x1875062187622016/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201762.436660:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187622016/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201762.436673:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201762.436677:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201762.436680:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187622016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201762.436686:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201762/real 1788201762] req@ffff8e828d723800 x1875062187622144/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201762.436694:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187622144/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201762.436698:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201762.436701:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201762.436703:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187622144:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201762.436706:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201762/real 1788201762] req@ffff8e828d721500 x1875062187622272/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201762.436714:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187622272/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201762.436719:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201762.436720:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201762.436722:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187622272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201762.436726:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201762/real 1788201762] req@ffff8e828d720000 x1875062187621760/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201762.436733:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187621760/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201762.436738:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201762.436741:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201762.436743:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187621760:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788201762.436747:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201762/real 1788201762] req@ffff8e828d720e00 x1875062187621888/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201762.436753:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187621888/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201762.436758:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201762.436760:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201762.436762:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187621888:192.168.201.121@tcp:38:kworker.0 00010000:00010000:0.0:1788201763.012544:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201764.036436:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201765.060452:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201766.084451:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201767.108435:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201767.428491:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201767.428502:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201767.428507:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201767.428519:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3222 00000100:00080000:3.0:1788201767.428530:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201767.428539:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788201767.428579:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187622400/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201767.428615:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201767.428620:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201767.428622:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201767.428627:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3222 00000100:00080000:3.0:1788201767.428631:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201767.428636:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201767.428650:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201767.428665:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187622528/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201767.428670:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187622400:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788201767.428688:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201767.428692:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201767.428695:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201767.428700:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3222 00000100:00080000:3.0:1788201767.428704:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201767.428709:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201767.428715:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201767.428728:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187622656/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201767.428738:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201767.428742:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201767.428744:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201767.428748:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3222 00000100:00080000:3.0:1788201767.428752:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201767.428757:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201767.428763:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201767.428773:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187622528:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201767.428779:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187622784/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201767.428801:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788201767.428803:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187622656:192.168.201.121@tcp:38:kworker.0 00000100:00080000:3.0:1788201767.428809:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201767.428813:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201767.428819:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3222 00000100:00080000:3.0:1788201767.428833:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201767.428839:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788201767.428842:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187622784:192.168.201.121@tcp:8:kworker.0 00000008:00100000:3.0:1788201767.428847:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788201767.428864:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187622912/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201767.428886:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187622912:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201768.132436:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201769.156517:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201770.180472:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788201770.500591:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.201.121:988 00000400:00000100:1.0:1788201770.500627:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788201770.500640:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788201770.500675:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788201770.500685:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788201770.500697:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788201770.500706:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:2.0:1788201770.500725:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201767/real 1788201770] req@ffff8e8289538380 x1875062187622784/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201770.500760:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187622784/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201770.500772:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201770.500775:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201770.500779:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187622784:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201770.500785:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201767/real 1788201770] req@ffff8e828953b100 x1875062187622528/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201770.500797:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187622528/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201770.500806:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201770.500810:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201770.500813:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187622528:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201770.500821:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201767/real 1788201770] req@ffff8e828953bb80 x1875062187622656/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201770.500833:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187622656/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201770.500840:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201770.500843:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201770.500847:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187622656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201770.500853:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201767/real 1788201770] req@ffff8e8289538a80 x1875062187622400/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201770.500868:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187622400/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201770.500876:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201770.500879:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201770.500883:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187622400:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201770.500910:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201767/real 1788201770] req@ffff8e828953a680 x1875062187622912/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201770.500923:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187622912/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201770.500930:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201770.500933:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201770.500936:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187622912:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201771.204496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201772.228549:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201772.548437:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201772.548449:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201772.548455:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201772.548467:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3227 00000100:00080000:3.0:1788201772.548478:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201772.548487:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788201772.548527:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187623040/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201772.548558:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201772.548562:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201772.548564:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201772.548569:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3227 00000100:00080000:3.0:1788201772.548574:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201772.548579:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788201772.548587:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187623040:192.168.201.121@tcp:250:kworker.0 00000008:00100000:3.0:1788201772.548604:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201772.548626:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187623168/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201772.548639:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201772.548643:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201772.548645:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788201772.548651:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788201772.548655:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3227 00000100:00080000:3.0:1788201772.548659:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201772.548666:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201772.548672:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201772.548685:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187623296/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201772.548695:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201772.548705:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201772.548707:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788201772.548715:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201772/real 1788201772] req@ffff8e8289539f80 x1875062187623040/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201772.548718:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3227 00000100:00080000:3.0:1788201772.548726:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201772.548734:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201772.548741:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201772.548752:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187623040/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201772.548759:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187623424/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201772.548771:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201772.548774:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201772.548780:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788201772.548781:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:3.0:1788201772.548785:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788201772.548791:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187623040:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788201772.548794:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3227 00000100:00080000:3.0:1788201772.548801:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201772.548807:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201772.548813:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201772.548826:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187623168:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201772.548827:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187623552/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201772.548862:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201772.548879:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201772/real 1788201772] req@ffff8e8289539880 x1875062187623168/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201772.548891:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187623168/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201772.548899:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201772.548902:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201772.548905:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187623168:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201772.548911:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187623296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201772.548931:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201772.548943:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201772/real 1788201772] req@ffff8e8289539500 x1875062187623296/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201772.548955:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187623296/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201772.548963:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201772.548967:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201772.548970:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187623296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201772.548977:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187623424:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201772.548997:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201772.549009:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201772/real 1788201772] req@ffff8e828953a680 x1875062187623424/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201772.549020:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187623424/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201772.549028:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201772.549032:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201772.549035:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187623424:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201772.549051:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187623552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201772.549071:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201772.549083:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201772/real 1788201772] req@ffff8e8289538a80 x1875062187623552/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201772.549098:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187623552/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201772.549105:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201772.549108:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201772.549111:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187623552:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201773.252463:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201774.277427:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201775.300427:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201776.324477:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201777.348438:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201777.668425:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201777.668433:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201777.668437:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201777.668446:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3232 00000100:00080000:3.0:1788201777.668456:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201777.668463:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788201777.668495:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187623680/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201777.668526:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201777.668529:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201777.668530:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201777.668533:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3232 00000100:00080000:3.0:1788201777.668535:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201777.668538:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201777.668546:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201777.668556:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187623808/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201777.668570:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201777.668572:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201777.668573:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201777.668576:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3232 00000100:00080000:3.0:1788201777.668577:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201777.668579:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201777.668583:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201777.668589:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187623936/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201777.668597:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201777.668599:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201777.668600:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201777.668608:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3232 00000100:00100000:2.0:1788201777.668610:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187623680:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788201777.668612:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201777.668624:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201777.668631:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788201777.668638:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187624064/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201777.668644:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201777.668646:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201777.668646:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201777.668649:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3232 00000100:00080000:3.0:1788201777.668651:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201777.668655:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201777.668658:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788201777.668683:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187624192/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201777.668712:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201777.668775:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201777/real 1788201777] req@ffff8e828953bb80 x1875062187623680/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201777.668805:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187623680/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201777.668820:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201777.668823:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201777.668826:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187623680:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201777.668833:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187623808:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201777.668851:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201777.668861:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201777/real 1788201777] req@ffff8e828953b100 x1875062187623808/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201777.668869:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187623808/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201777.668875:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201777.668877:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201777.668880:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187623808:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201777.668904:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187623936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201777.668916:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201777.668922:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201777/real 1788201777] req@ffff8e8289538380 x1875062187623936/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201777.668937:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187623936/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201777.668943:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201777.668945:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201777.668947:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187623936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201777.668952:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187624064:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201777.668966:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201777.668973:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201777/real 1788201777] req@ffff8e8289538a80 x1875062187624064/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201777.668980:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187624064/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201777.668985:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201777.668987:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201777.668989:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187624064:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201777.668993:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187624192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201777.669003:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201777.669009:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201777/real 1788201777] req@ffff8e828953a680 x1875062187624192/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201777.669016:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187624192/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201777.669028:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201777.669030:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201777.669032:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187624192:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201778.372478:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201779.396465:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201780.420505:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201781.444486:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201782.468476:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201782.788370:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201782.788379:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201782.788382:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201782.788390:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3237 00000100:00080000:3.0:1788201782.788399:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201782.788405:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788201782.788438:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187624320/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201782.788461:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201782.788464:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201782.788465:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201782.788468:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3237 00000100:00080000:3.0:1788201782.788470:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201782.788472:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201782.788479:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201782.788488:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187624448/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201782.788495:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201782.788499:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201782.788500:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201782.788503:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3237 00000100:00080000:3.0:1788201782.788505:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201782.788507:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201782.788509:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201782.788514:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187624576/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201782.788520:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201782.788522:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201782.788523:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201782.788526:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3237 00000100:00080000:3.0:1788201782.788529:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201782.788532:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201782.788535:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788201782.788541:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187624704/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201782.788548:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788201782.788549:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187624320:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788201782.788551:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201782.788555:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201782.788559:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3237 00000100:00080000:3.0:1788201782.788562:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201782.788564:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201782.788567:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788201782.788576:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187624832/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201782.788646:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187624448:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201782.788675:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187624576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201782.788701:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187624704:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201782.788720:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187624832:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201783.492443:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201784.516483:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201785.540439:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788201785.668583:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.201.121:988 00000400:00000100:1.0:1788201785.668619:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788201785.668633:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788201785.668664:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788201785.668670:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788201785.668675:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788201785.668680:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:2.0:1788201785.668721:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201782/real 1788201785] req@ffff8e828953a680 x1875062187624704/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201785.668748:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187624704/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201785.668760:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201785.668765:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201785.668769:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187624704:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201785.668783:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201782/real 1788201785] req@ffff8e8289538a80 x1875062187624832/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201785.668797:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187624832/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201785.668805:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201785.668808:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201785.668812:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187624832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201785.668819:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201782/real 1788201785] req@ffff8e8289539500 x1875062187624320/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201785.668832:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187624320/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201785.668841:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201785.668844:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201785.668848:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187624320:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201785.668857:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201782/real 1788201785] req@ffff8e8289539880 x1875062187624448/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201785.668868:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187624448/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201785.668875:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201785.668878:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201785.668881:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187624448:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201785.668888:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201782/real 1788201785] req@ffff8e8289539f80 x1875062187624576/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201785.668911:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187624576/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201785.668919:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201785.668923:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201785.668926:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187624576:192.168.201.121@tcp:38:kworker.0 00010000:00010000:1.0:1788201786.564470:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201787.588490:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201787.908370:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201787.908379:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201787.908383:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201787.908391:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3242 00000100:00080000:3.0:1788201787.908399:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201787.908405:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788201787.908436:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187624960/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201787.908459:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201787.908462:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201787.908464:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201787.908466:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3242 00000100:00080000:3.0:1788201787.908469:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201787.908471:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201787.908478:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201787.908487:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187625088/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201787.908495:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201787.908498:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201787.908499:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201787.908501:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3242 00000100:00080000:3.0:1788201787.908504:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201787.908506:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201787.908510:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201787.908517:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187624960:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788201787.908518:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187625216/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201787.908532:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201787.908534:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201787.908535:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201787.908538:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3242 00000100:00080000:3.0:1788201787.908540:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201787.908542:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201787.908546:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788201787.908552:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187625344/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201787.908559:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201787.908561:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201787.908562:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201787.908564:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3242 00000100:00080000:3.0:1788201787.908575:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201787.908578:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201787.908582:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788201787.908590:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187625472/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201787.908593:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201787.908632:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201787/real 1788201787] req@ffff8e8289538380 x1875062187624960/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201787.908646:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187624960/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201787.908654:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201787.908657:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201787.908660:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187624960:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201787.908665:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187625088:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201787.908680:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201787.908688:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201787/real 1788201787] req@ffff8e828953b100 x1875062187625088/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201787.908696:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187625088/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201787.908701:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201787.908703:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201787.908705:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187625088:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201787.908722:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187625216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201787.908733:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201787.908739:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201787/real 1788201787] req@ffff8e828953bb80 x1875062187625216/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201787.908746:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187625216/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201787.908756:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201787.908758:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201787.908760:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187625216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201787.908763:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187625344:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201787.908773:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201787.908778:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201787/real 1788201787] req@ffff8e8289539f80 x1875062187625344/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201787.908785:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187625344/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201787.908790:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201787.908792:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201787.908794:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187625344:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201787.908797:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187625472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201787.908813:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201787.908839:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201787/real 1788201787] req@ffff8e8289539880 x1875062187625472/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201787.908852:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187625472/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201787.908876:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201787.908880:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201787.908885:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187625472:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201788.612420:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201789.636432:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201790.660480:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201791.684470:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201792.708437:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201793.028485:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201793.028493:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201793.028497:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201793.028505:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3248 00000100:00080000:2.0:1788201793.028515:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201793.028522:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788201793.028569:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187625600/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201793.028600:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201793.028604:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201793.028606:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201793.028611:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3248 00000100:00080000:2.0:1788201793.028615:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201793.028621:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201793.028630:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201793.028634:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187625600:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201793.028647:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187625728/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201793.028661:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201793.028665:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201793.028667:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201793.028671:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3248 00000100:00080000:2.0:1788201793.028685:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201793.028690:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:3.0:1788201793.028695:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:2.0:1788201793.028696:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201793.028712:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187625856/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201793.028724:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201793.028728:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788201793.028730:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201793.028734:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3248 00000100:00080000:2.0:1788201793.028738:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201793.028743:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201793.028749:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788201793.028756:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201793/real 1788201793] req@ffff8e82894e5880 x1875062187625600/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201793.028765:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187625984/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201793.028795:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788201793.028799:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187625600/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201793.028801:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788201793.028805:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201793.028816:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3248 00000100:00080000:3.0:1788201793.028819:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201793.028822:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201793.028827:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:2.0:1788201793.028831:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788201793.028837:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187625600:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788201793.028840:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201793.028856:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187626112/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201793.028867:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187625728:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201793.028897:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201793.028908:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201793/real 1788201793] req@ffff8e82894e4000 x1875062187625728/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201793.028917:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187625728/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201793.028923:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201793.028925:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201793.028927:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187625728:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201793.028931:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187625856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201793.028944:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201793.028951:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201793/real 1788201793] req@ffff8e82894e6a00 x1875062187625856/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201793.028959:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187625856/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201793.028963:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201793.028965:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201793.028967:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187625856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201793.028971:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187625984:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201793.028981:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201793.028987:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201793/real 1788201793] req@ffff8e82894e6680 x1875062187625984/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201793.028994:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187625984/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201793.028998:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201793.029000:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201793.029002:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187625984:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201793.029012:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187626112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201793.029021:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201793.029027:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201793/real 1788201793] req@ffff8e82894e7480 x1875062187626112/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201793.029033:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187626112/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201793.029039:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201793.029041:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201793.029043:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187626112:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201793.732432:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201794.756448:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201795.780457:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201796.804487:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201797.828534:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201798.148416:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201798.148424:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201798.148428:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201798.148437:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3253 00000100:00080000:2.0:1788201798.148446:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201798.148453:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788201798.148489:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187626240/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201798.148520:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201798.148525:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201798.148527:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201798.148532:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3253 00000100:00080000:2.0:1788201798.148535:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201798.148538:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201798.148547:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201798.148558:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187626368/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201798.148568:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187626240:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788201798.148570:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201798.148577:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201798.148579:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201798.148582:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3253 00000100:00080000:2.0:1788201798.148584:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201798.148587:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201798.148591:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201798.148604:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187626496/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201798.148615:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201798.148619:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788201798.148621:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201798.148625:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3253 00000100:00080000:2.0:1788201798.148631:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788201798.148634:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788201798.148642:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201798.148648:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201798.148661:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187626624/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201798.148671:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201798.148675:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788201798.148680:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788201798.148692:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201798/real 1788201798] req@ffff8e82894e4e00 x1875062187626240/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201798.148702:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3253 00000100:00080000:2.0:1788201798.148709:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201798.148715:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788201798.148718:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187626240/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788201798.148722:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788201798.148738:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1788201798.148740:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187626752/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201798.148746:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201798.148756:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187626240:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788201798.148782:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187626368:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201798.148808:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201798.148822:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201798/real 1788201798] req@ffff8e82894e7800 x1875062187626368/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201798.148835:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187626368/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201798.148844:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201798.148847:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201798.148851:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187626368:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201798.148858:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187626496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201798.148878:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201798.148887:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201798/real 1788201798] req@ffff8e82894e6300 x1875062187626496/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201798.148895:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187626496/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201798.148900:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201798.148903:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201798.148905:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187626496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201798.148909:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187626624:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201798.148923:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201798.148933:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201798/real 1788201798] req@ffff8e82894e4a80 x1875062187626624/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201798.148946:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187626624/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201798.148954:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201798.148957:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201798.148961:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187626624:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201798.148967:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187626752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201798.148983:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201798.148991:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201798/real 1788201798] req@ffff8e82894e5c00 x1875062187626752/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201798.149007:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187626752/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201798.149013:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201798.149015:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201798.149017:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187626752:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201798.852506:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201799.876445:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201800.900484:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201801.924504:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201802.948496:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201803.268405:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201803.268414:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201803.268418:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201803.268425:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3258 00000100:00080000:0.0:1788201803.268434:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201803.268440:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201803.268470:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187626880/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201803.268497:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201803.268499:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201803.268501:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201803.268504:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3258 00000100:00080000:0.0:1788201803.268506:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201803.268508:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201803.268516:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201803.268526:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187627008/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201803.268534:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187626880:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788201803.268536:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201803.268543:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201803.268544:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201803.268547:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3258 00000100:00080000:0.0:1788201803.268549:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201803.268552:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201803.268555:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201803.268562:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187627136/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201803.268568:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201803.268571:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201803.268572:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201803.268575:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3258 00000100:00080000:0.0:1788201803.268578:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201803.268580:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201803.268583:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201803.268590:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187627264/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201803.268609:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201803.268611:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201803.268613:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201803.268615:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3258 00000100:00080000:0.0:1788201803.268618:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201803.268620:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201803.268624:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201803.268632:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187627392/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201803.268661:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187627008:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201803.268689:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187627136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201803.268703:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187627264:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201803.268715:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187627392:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201803.972450:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201804.996449:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788201805.060605:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:2.0:1788201805.060642:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788201805.060654:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788201805.060683:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788201805.060696:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788201805.060708:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:3.0:1788201805.060714:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201803/real 1788201805] req@ffff8e828793f480 x1875062187627008/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1788201805.060723:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:3.0:1788201805.060744:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187627008/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201805.060754:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201805.060765:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201805.060768:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187627008:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201805.060774:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201803/real 1788201805] req@ffff8e828793ca80 x1875062187627136/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201805.060787:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187627136/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201805.060792:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201805.060794:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201805.060796:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187627136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201805.060800:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201803/real 1788201805] req@ffff8e828793c700 x1875062187627264/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201805.060807:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187627264/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201805.060811:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201805.060813:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201805.060815:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187627264:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201805.060819:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201803/real 1788201805] req@ffff8e828793f100 x1875062187627392/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201805.060826:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187627392/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201805.060830:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201805.060832:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201805.060834:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187627392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201805.060837:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201803/real 1788201805] req@ffff8e828793ce00 x1875062187626880/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201805.060845:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187626880/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201805.060849:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201805.060851:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201805.060853:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187626880:192.168.201.121@tcp:250:kworker.0 00010000:00010000:0.0:1788201806.020476:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201807.044584:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201808.068470:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201808.388467:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201808.388477:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201808.388482:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201808.388493:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3263 00000100:00080000:0.0:1788201808.388522:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201808.388542:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201808.388588:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187627520/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201808.388614:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201808.388617:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201808.388619:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201808.388622:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3263 00000100:00080000:0.0:1788201808.388629:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201808.388632:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201808.388654:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201808.388662:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187627648/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201808.388670:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201808.388674:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201808.388676:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201808.388681:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3263 00000100:00080000:0.0:1788201808.388684:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201808.388686:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201808.388690:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201808.388696:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187627776/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201808.388702:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201808.388704:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201808.388705:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201808.388707:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3263 00000100:00080000:0.0:1788201808.388710:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201808.388714:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201808.388722:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201808.388734:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187627520:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788201808.388743:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187627904/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201808.388757:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201808.388762:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201808.388764:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201808.388769:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3263 00000100:00080000:0.0:1788201808.388773:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201808.388776:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201808.388782:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201808.388792:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187628032/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201808.388829:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201808.388881:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201808/real 1788201808] req@ffff8e828793e300 x1875062187627520/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201808.388896:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187627520/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201808.388906:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201808.388910:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201808.388913:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187627520:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201808.388919:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187627648:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201808.388937:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201808.388945:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201808/real 1788201808] req@ffff8e828793e680 x1875062187627648/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201808.388953:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187627648/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201808.388958:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201808.388961:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201808.388963:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187627648:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201808.388966:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187627776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201808.388977:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201808.388983:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201808/real 1788201808] req@ffff8e828793f800 x1875062187627776/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201808.388991:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187627776/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201808.388995:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201808.388997:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201808.388999:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187627776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201808.389019:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187627904:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201808.389031:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201808.389038:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201808/real 1788201808] req@ffff8e828793ce00 x1875062187627904/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201808.389045:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187627904/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201808.389050:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201808.389052:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201808.389053:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187627904:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201808.389057:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187628032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201808.389067:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201808.389073:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201808/real 1788201808] req@ffff8e828793f100 x1875062187628032/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201808.389084:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187628032/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201808.389089:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201808.389091:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201808.389093:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187628032:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201809.092488:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201810.116434:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201811.140496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201812.164501:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201813.188448:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201813.508447:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201813.508472:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201813.508475:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201813.508484:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3268 00000100:00080000:3.0:1788201813.508493:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201813.508500:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788201813.508533:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187628160/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201813.508557:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201813.508560:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201813.508561:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201813.508564:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3268 00000100:00080000:3.0:1788201813.508566:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201813.508569:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201813.508576:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201813.508585:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187628288/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201813.508599:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201813.508602:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201813.508603:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201813.508606:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3268 00000100:00080000:3.0:1788201813.508608:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788201813.508612:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187628160:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788201813.508613:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201813.508621:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201813.508628:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187628416/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201813.508636:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201813.508639:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201813.508640:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201813.508643:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3268 00000100:00080000:3.0:1788201813.508645:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201813.508648:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201813.508651:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788201813.508660:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187628544/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201813.508684:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201813.508686:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201813.508687:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201813.508689:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3268 00000100:00080000:3.0:1788201813.508693:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201813.508695:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201813.508699:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788201813.508703:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201813.508710:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187628672/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201813.508748:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201813/real 1788201813] req@ffff8e8289539500 x1875062187628160/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201813.508764:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187628160/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201813.508772:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201813.508775:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201813.508778:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187628160:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201813.508784:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187628288:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201813.508800:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201813.508808:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201813/real 1788201813] req@ffff8e8289538a80 x1875062187628288/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201813.508816:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187628288/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201813.508825:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201813.508828:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201813.508830:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187628288:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201813.508848:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187628416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201813.508858:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201813.508865:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201813/real 1788201813] req@ffff8e828953a680 x1875062187628416/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201813.508872:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187628416/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201813.508877:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201813.508879:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201813.508881:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187628416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201813.508884:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187628544:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201813.508894:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201813.508900:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201813/real 1788201813] req@ffff8e8289539880 x1875062187628544/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201813.508907:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187628544/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201813.508912:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201813.508914:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201813.508916:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187628544:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201813.508920:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187628672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201813.508930:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201813.508935:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201813/real 1788201813] req@ffff8e8289539f80 x1875062187628672/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201813.508942:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187628672/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201813.508946:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201813.508948:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201813.508950:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187628672:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201814.212448:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201815.236491:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201816.260490:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201817.284496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201818.308467:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201818.628426:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201818.628433:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201818.628437:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201818.628446:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3273 00000100:00080000:0.0:1788201818.628455:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201818.628462:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201818.628494:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187628800/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201818.628522:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201818.628526:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201818.628538:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201818.628543:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3273 00000100:00080000:0.0:1788201818.628548:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201818.628552:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201818.628561:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201818.628575:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187628800:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788201818.628579:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187628928/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201818.628592:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201818.628596:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201818.628599:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201818.628603:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3273 00000100:00080000:0.0:1788201818.628608:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201818.628621:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201818.628628:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788201818.628634:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201818.628643:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187629056/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201818.628653:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201818.628656:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201818.628657:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201818.628661:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3273 00000100:00080000:0.0:1788201818.628665:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788201818.628669:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201818/real 1788201818] req@ffff8e828793c700 x1875062187628800/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201818.628671:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201818.628679:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201818.628689:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187628800/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201818.628691:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187629184/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201818.628715:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201818.628716:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201818.628722:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201818.628724:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201818.628724:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:0.0:1788201818.628733:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3273 00000100:00100000:2.0:1788201818.628734:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187628800:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788201818.628736:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201818.628743:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201818.628747:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201818.628754:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187629312/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201818.628760:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187628928:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201818.628780:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201818.628790:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201818/real 1788201818] req@ffff8e828793ca80 x1875062187628928/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201818.628798:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187628928/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201818.628804:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201818.628806:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201818.628808:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187628928:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201818.628812:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187629056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201818.628822:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201818.628827:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201818/real 1788201818] req@ffff8e828793f480 x1875062187629056/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201818.628835:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187629056/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201818.628840:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201818.628842:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201818.628844:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187629056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201818.628847:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187629184:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201818.628857:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201818.628863:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201818/real 1788201818] req@ffff8e828793c000 x1875062187629184/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201818.628870:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187629184/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201818.628875:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201818.628878:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201818.628880:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187629184:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201818.628888:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187629312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201818.628898:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201818.628905:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201818/real 1788201818] req@ffff8e828793dc00 x1875062187629312/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201818.628912:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187629312/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201818.628917:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201818.628918:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201818.628920:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187629312:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201819.332480:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201820.356510:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201821.380477:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201822.404565:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201823.428573:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201823.748407:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201823.748415:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201823.748418:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201823.748426:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3278 00000100:00080000:0.0:1788201823.748436:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201823.748443:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201823.748483:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793df80 x1875062187629440/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201823.748512:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201823.748517:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201823.748519:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201823.748524:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3278 00000100:00080000:0.0:1788201823.748529:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201823.748533:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201823.748543:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201823.748558:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187629568/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201823.748573:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201823.748576:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201823.748578:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201823.748581:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3278 00000100:00080000:0.0:1788201823.748584:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201823.748590:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788201823.748596:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187629440:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788201823.748600:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201823.748621:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187629696/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201823.748635:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201823.748639:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201823.748641:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201823.748645:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3278 00000100:00080000:0.0:1788201823.748650:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201823.748653:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201823.748659:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201823.748668:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187629824/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201823.748676:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201823.748679:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201823.748680:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201823.748686:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3278 00000800:00000100:2.0:1788201823.748694:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788201823.748698:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201823.748703:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201823.748707:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201823.748719:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187629952/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201823.748760:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201823/real 1788201823] req@ffff8e828793df80 x1875062187629440/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201823.748784:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793df80 x1875062187629440/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201823.748798:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201823.748805:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201823.748810:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187629440:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201823.748819:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187629568:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201823.748842:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201823.748853:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201823/real 1788201823] req@ffff8e828793d500 x1875062187629568/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201823.748863:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187629568/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201823.748869:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201823.748871:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201823.748874:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187629568:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201823.748897:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187629696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201823.748910:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201823.748917:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201823/real 1788201823] req@ffff8e828793d880 x1875062187629696/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201823.748929:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187629696/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201823.748938:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201823.748942:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201823.748945:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187629696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201823.748952:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187629824:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201823.748973:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201823.748982:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201823/real 1788201823] req@ffff8e828793c380 x1875062187629824/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201823.748999:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187629824/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201823.749005:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201823.749007:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201823.749009:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187629824:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201823.749014:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187629952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201823.749028:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201823.749035:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201823/real 1788201823] req@ffff8e828793fb80 x1875062187629952/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201823.749043:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187629952/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201823.749049:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201823.749050:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201823.749052:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187629952:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201824.452581:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201825.476524:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201826.500519:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201827.524475:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201828.548582:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201828.868379:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201828.868388:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201828.868391:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201828.868400:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3283 00000100:00080000:0.0:1788201828.868408:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201828.868416:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201828.868453:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187630080/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201828.868485:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201828.868488:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201828.868489:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201828.868493:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3283 00000100:00080000:0.0:1788201828.868496:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201828.868499:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201828.868509:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201828.868524:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187630208/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201828.868531:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187630080:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788201828.868540:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201828.868545:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201828.868546:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201828.868551:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3283 00000100:00080000:0.0:1788201828.868555:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201828.868560:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201828.868565:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201828.868579:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187630336/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201828.868590:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201828.868593:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201828.868595:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201828.868600:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3283 00000800:00000100:2.0:1788201828.868605:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788201828.868607:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201828.868615:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201828.868621:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201828.868634:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187630464/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201828.868645:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201828.868649:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201828.868651:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201828.868654:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3283 00000100:00080000:0.0:1788201828.868658:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201828.868663:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788201828.868667:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201828/real 1788201828] req@ffff8e828793fb80 x1875062187630080/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788201828.868670:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201828.868690:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793df80 x1875062187630592/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201828.868697:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187630080/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201828.868742:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201828.868748:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201828.868754:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187630080:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201828.868780:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187630208:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201828.868806:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201828.868820:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201828/real 1788201828] req@ffff8e828793c380 x1875062187630208/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201828.868833:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187630208/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201828.868842:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201828.868846:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201828.868850:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187630208:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201828.868857:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187630336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201828.868878:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201828.868891:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201828/real 1788201828] req@ffff8e828793d880 x1875062187630336/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201828.868904:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187630336/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201828.868913:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201828.868917:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201828.868920:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187630336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201828.868927:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187630464:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201828.868949:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201828.868961:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201828/real 1788201828] req@ffff8e828793d500 x1875062187630464/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201828.868974:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187630464/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201828.868983:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201828.868986:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201828.868990:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187630464:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201828.868996:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187630592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201828.869034:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201828.869048:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201828/real 1788201828] req@ffff8e828793df80 x1875062187630592/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201828.869060:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793df80 x1875062187630592/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201828.869069:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201828.869072:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201828.869076:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187630592:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201829.572543:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201830.596491:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201831.620531:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201832.644493:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201833.668523:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201833.988445:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201833.988456:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201833.988460:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201833.988470:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3289 00000100:00080000:1.0:1788201833.988481:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201833.988489:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788201833.988526:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187630720/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201833.988557:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201833.988561:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201833.988563:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201833.988568:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3289 00000100:00080000:1.0:1788201833.988577:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201833.988581:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201833.988591:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201833.988607:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187630720:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788201833.988610:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187630848/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201833.988626:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201833.988630:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788201833.988632:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201833.988636:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3289 00000100:00080000:1.0:1788201833.988640:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201833.988645:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201833.988650:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201833.988662:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187630976/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201833.988676:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788201833.988695:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201833.988699:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201833.988701:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201833.988706:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3289 00000100:00080000:1.0:1788201833.988711:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201833.988715:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201833.988721:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201833.988728:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201833/real 1788201833] req@ffff8e828d720380 x1875062187630720/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201833.988739:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187631104/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201833.988751:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788201833.988752:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187630720/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201833.988760:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201833.988765:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201833.988768:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201833.988773:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3289 00000100:00080000:2.0:1788201833.988776:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:1.0:1788201833.988780:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788201833.988781:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187630720:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788201833.988788:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201833.988795:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201833.988804:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187630848:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788201833.988820:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187631232/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201833.988830:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201833.988844:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201833/real 1788201833] req@ffff8e828d722300 x1875062187630848/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201833.988853:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187630848/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201833.988860:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201833.988863:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201833.988865:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187630848:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201833.988869:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187630976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201833.988879:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201833.988885:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201833/real 1788201833] req@ffff8e828d722d80 x1875062187630976/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201833.988892:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187630976/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201833.988902:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201833.988904:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201833.988906:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187630976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201833.988909:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187631104:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201833.988919:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201833.988925:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201833/real 1788201833] req@ffff8e828d721180 x1875062187631104/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201833.988931:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187631104/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201833.988936:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201833.988938:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201833.988940:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187631104:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201833.988951:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187631232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201833.988960:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201833.988966:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201833/real 1788201833] req@ffff8e828d720a80 x1875062187631232/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201833.988973:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187631232/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201833.988978:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201833.988980:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201833.988982:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187631232:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201834.692480:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201835.716521:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201836.740479:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201837.764478:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201838.788501:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201839.108421:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201839.108430:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201839.108434:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201839.108441:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3294 00000100:00080000:2.0:1788201839.108451:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201839.108457:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788201839.108489:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187631360/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201839.108514:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201839.108517:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201839.108518:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201839.108521:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3294 00000100:00080000:2.0:1788201839.108524:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201839.108526:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201839.108533:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201839.108542:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187631488/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201839.108549:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201839.108552:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201839.108553:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788201839.108560:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187631360:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788201839.108563:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3294 00000100:00080000:2.0:1788201839.108569:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201839.108572:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201839.108575:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201839.108584:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187631616/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201839.108591:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201839.108594:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788201839.108595:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201839.108598:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3294 00000100:00080000:2.0:1788201839.108601:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201839.108603:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201839.108606:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201839.108612:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187631744/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201839.108618:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201839.108620:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788201839.108621:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201839.108623:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3294 00000100:00080000:2.0:1788201839.108626:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201839.108628:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201839.108631:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201839.108638:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187631872/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201839.108665:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187631488:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201839.108701:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187631616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201839.108723:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187631744:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788201839.108758:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187631872:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201839.812465:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201840.836496:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201841.860505:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788201842.180515:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788201842.180565:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788201842.180584:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788201842.180620:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788201842.180631:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788201842.180642:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788201842.180651:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:0.0:1788201842.180678:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201839/real 1788201842] req@ffff8e82894e7100 x1875062187631616/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201842.180706:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187631616/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201842.180718:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201842.180723:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201842.180728:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187631616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201842.180736:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201839/real 1788201842] req@ffff8e82894e4380 x1875062187631744/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201842.180749:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187631744/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201842.180757:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201842.180761:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201842.180764:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187631744:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788201842.180771:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201839/real 1788201842] req@ffff8e82894e5c00 x1875062187631872/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201842.180782:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187631872/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201842.180790:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201842.180794:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201842.180798:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187631872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788201842.180804:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201839/real 1788201842] req@ffff8e82894e5f80 x1875062187631360/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201842.180815:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187631360/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201842.180823:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201842.180826:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201842.180829:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187631360:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788201842.180836:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201839/real 1788201842] req@ffff8e82894e6d80 x1875062187631488/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201842.180847:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187631488/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201842.180854:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201842.180858:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201842.180862:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187631488:192.168.201.121@tcp:38:kworker.0 00010000:00010000:1.0:1788201842.884489:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201843.908495:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201844.228435:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201844.228444:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201844.228448:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201844.228456:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3299 00000100:00080000:2.0:1788201844.228465:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201844.228471:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788201844.228511:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187632000/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201844.228535:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201844.228538:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201844.228540:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201844.228543:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3299 00000100:00080000:2.0:1788201844.228545:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201844.228547:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201844.228555:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201844.228567:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187632128/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201844.228577:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201844.228581:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201844.228583:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201844.228587:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3299 00000100:00080000:2.0:1788201844.228590:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201844.228594:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201844.228600:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201844.228619:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187632256/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201844.228629:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201844.228632:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788201844.228634:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201844.228638:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3299 00000100:00080000:2.0:1788201844.228642:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201844.228646:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201844.228653:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201844.228661:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187632000:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201844.228671:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187632384/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201844.228687:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201844.228691:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788201844.228692:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201844.228697:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3299 00000100:00080000:2.0:1788201844.228702:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201844.228706:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201844.228711:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201844.228730:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187632512/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788201844.228783:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201844.228866:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201844/real 1788201844] req@ffff8e82894e4a80 x1875062187632000/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201844.228920:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187632000/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201844.228934:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201844.228939:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201844.228944:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187632000:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788201844.228953:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187632128:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788201844.228978:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201844.228993:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201844/real 1788201844] req@ffff8e82894e6300 x1875062187632128/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201844.229005:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187632128/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201844.229013:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201844.229016:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201844.229019:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187632128:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201844.229026:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187632256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788201844.229048:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201844.229060:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201844/real 1788201844] req@ffff8e82894e7800 x1875062187632256/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201844.229073:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187632256/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201844.229081:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201844.229085:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201844.229088:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187632256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201844.229123:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187632384:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788201844.229147:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201844.229160:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201844/real 1788201844] req@ffff8e82894e4e00 x1875062187632384/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201844.229173:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187632384/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201844.229182:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201844.229185:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201844.229188:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187632384:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788201844.229194:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187632512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788201844.229213:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201844.229224:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201844/real 1788201844] req@ffff8e82894e7480 x1875062187632512/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201844.229242:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187632512/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201844.229250:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201844.229253:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201844.229257:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187632512:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201844.932474:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201845.956435:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201846.980425:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201848.004471:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201849.028492:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201849.348423:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201849.348432:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201849.348436:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201849.348445:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3304 00000100:00080000:0.0:1788201849.348455:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201849.348462:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201849.348493:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187632640/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201849.348516:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201849.348518:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201849.348520:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201849.348523:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3304 00000100:00080000:0.0:1788201849.348525:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201849.348528:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201849.348536:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201849.348545:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187632768/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201849.348549:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187632640:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788201849.348560:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201849.348563:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201849.348564:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201849.348567:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3304 00000100:00080000:0.0:1788201849.348569:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201849.348571:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201849.348574:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201849.348582:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187632896/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201849.348589:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201849.348591:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201849.348592:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201849.348595:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3304 00000100:00080000:0.0:1788201849.348599:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201849.348602:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201849.348605:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201849.348611:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187633024/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201849.348618:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201849.348620:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201849.348621:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201849.348626:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3304 00000800:00000100:2.0:1788201849.348627:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788201849.348630:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201849.348633:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201849.348636:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201849.348644:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187633152/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201849.348684:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201849/real 1788201849] req@ffff8e828793dc00 x1875062187632640/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201849.348706:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187632640/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201849.348715:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201849.348720:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201849.348724:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187632640:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201849.348748:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187632768:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201849.348783:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201849.348818:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201849/real 1788201849] req@ffff8e828793c000 x1875062187632768/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201849.348828:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187632768/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201849.348834:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201849.348836:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201849.348838:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187632768:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201849.348842:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187632896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201849.348858:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201849.348865:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201849/real 1788201849] req@ffff8e828793f480 x1875062187632896/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201849.348873:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187632896/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201849.348883:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201849.348885:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201849.348887:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187632896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201849.348890:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187633024:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201849.348902:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201849.348907:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201849/real 1788201849] req@ffff8e828793ca80 x1875062187633024/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201849.348915:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187633024/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201849.348920:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201849.348922:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201849.348924:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187633024:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201849.348927:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187633152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201849.348939:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201849.348945:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201849/real 1788201849] req@ffff8e828793c700 x1875062187633152/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201849.348952:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187633152/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201849.348957:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201849.348958:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201849.348961:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187633152:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201850.052447:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201851.076445:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201852.100483:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201853.124558:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201854.148501:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201854.468446:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201854.468456:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201854.468460:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201854.468471:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3309 00000100:00080000:3.0:1788201854.468482:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201854.468490:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788201854.468528:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187633280/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201854.468556:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201854.468561:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201854.468564:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201854.468569:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3309 00000100:00080000:3.0:1788201854.468575:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201854.468581:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201854.468591:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201854.468597:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187633280:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788201854.468610:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187633408/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201854.468625:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201854.468629:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201854.468631:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201854.468636:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3309 00000100:00080000:3.0:1788201854.468652:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201854.468658:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201854.468664:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788201854.468669:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201854.468682:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187633536/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201854.468700:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788201854.468729:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201854/real 1788201854] req@ffff8e828953bb80 x1875062187633280/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201854.468750:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187633280/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201854.468761:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201854.468765:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201854.468769:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187633280:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201854.468790:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187633408:192.168.201.121@tcp:38:kworker.0 00000100:00080000:3.0:1788201854.468803:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201854.468807:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201854.468813:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3309 00000100:00080000:3.0:1788201854.468819:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788201854.468824:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788201854.468828:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201854.468834:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201854.468845:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201854/real 1788201854] req@ffff8e828953b100 x1875062187633408/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201854.468849:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187633664/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201854.468887:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788201854.468891:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187633408/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201854.468891:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788201854.468903:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201854.468907:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201854.468912:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187633408:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201854.468920:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187633536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201854.468941:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201854.468951:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201854/real 1788201854] req@ffff8e8289538380 x1875062187633536/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201854.468961:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187633536/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201854.468967:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201854.468970:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201854.468972:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187633536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201854.468989:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187633664:192.168.201.121@tcp:8:kworker.0 00000100:00080000:3.0:1788201854.468992:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201854.468999:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3309 00000100:00080000:3.0:1788201854.469002:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201854.469005:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201854.469010:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788201854.469024:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187633792/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201854.469025:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201854.469050:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201854/real 1788201854] req@ffff8e828953bb80 x1875062187633664/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201854.469068:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187633664/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201854.469078:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201854.469081:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201854.469085:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187633664:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201854.469103:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187633792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201854.469125:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201854.469137:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201854/real 1788201854] req@ffff8e8289539f80 x1875062187633792/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201854.469150:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187633792/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201854.469160:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201854.469163:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201854.469167:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187633792:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201855.172505:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201856.196502:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201857.220463:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201858.244494:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201859.268458:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201859.588440:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201859.588451:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201859.588455:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201859.588465:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3314 00000100:00080000:2.0:1788201859.588476:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201859.588483:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788201859.588522:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187633920/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201859.588554:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201859.588557:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201859.588559:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201859.588569:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3314 00000100:00080000:2.0:1788201859.588573:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201859.588578:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201859.588587:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201859.588603:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187634048/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201859.588628:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201859.588631:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788201859.588633:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187633920:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788201859.588634:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201859.588642:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3314 00000100:00080000:2.0:1788201859.588646:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201859.588649:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201859.588653:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201859.588663:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187634176/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201859.588670:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201859.588672:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788201859.588674:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201859.588676:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3314 00000100:00080000:2.0:1788201859.588679:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201859.588682:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201859.588685:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201859.588691:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187634304/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201859.588697:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201859.588700:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788201859.588701:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788201859.588705:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788201859.588706:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3314 00000100:00080000:2.0:1788201859.588711:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201859.588714:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201859.588717:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201859.588727:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187634432/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201859.588748:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201859/real 1788201859] req@ffff8e82894e6680 x1875062187633920/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201859.588762:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187633920/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201859.588769:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201859.588772:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201859.588775:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187633920:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788201859.588793:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187634048:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788201859.588808:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201859.588817:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201859/real 1788201859] req@ffff8e82894e6a00 x1875062187634048/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201859.588824:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187634048/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201859.588830:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201859.588832:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201859.588835:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187634048:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201859.588838:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187634176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788201859.588848:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201859.588853:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201859/real 1788201859] req@ffff8e82894e4000 x1875062187634176/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201859.588860:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187634176/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201859.588865:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201859.588867:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201859.588869:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187634176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201859.588872:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187634304:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788201859.588882:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201859.588888:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201859/real 1788201859] req@ffff8e82894e5880 x1875062187634304/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201859.588895:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187634304/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201859.588899:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201859.588901:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201859.588904:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187634304:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788201859.588907:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187634432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788201859.588917:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201859.588922:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201859/real 1788201859] req@ffff8e82894e7480 x1875062187634432/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201859.588929:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187634432/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201859.588934:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201859.588935:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201859.588937:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187634432:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201860.292492:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201861.316449:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201862.340478:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201863.364478:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201864.388504:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201864.708434:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201864.708443:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201864.708447:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201864.708456:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3319 00000100:00080000:2.0:1788201864.708470:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201864.708478:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788201864.708518:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187634560/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201864.708557:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201864.708561:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201864.708565:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201864.708571:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3319 00000100:00080000:2.0:1788201864.708576:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201864.708582:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201864.708584:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187634560:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788201864.708591:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201864.708606:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187634688/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201864.708616:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201864.708620:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201864.708638:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201864.708642:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3319 00000100:00080000:2.0:1788201864.708647:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201864.708653:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201864.708660:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788201864.708672:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201864.708676:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187634816/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201864.708690:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201864.708694:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788201864.708696:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201864.708700:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3319 00000100:00080000:2.0:1788201864.708704:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201864.708708:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201864.708714:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201864.708720:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201864/real 1788201864] req@ffff8e82894e4e00 x1875062187634560/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201864.708727:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187634944/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201864.708762:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187634560/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201864.708770:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201864.708776:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788201864.708778:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201864.708779:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201864.708783:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3319 00000100:00080000:0.0:1788201864.708787:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201864.708790:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187634560:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788201864.708797:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201864.708805:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201864.708810:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201864.708816:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187634688:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201864.708827:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187635072/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788201864.708837:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201864.708852:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201864/real 1788201864] req@ffff8e82894e7800 x1875062187634688/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201864.708861:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187634688/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201864.708867:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201864.708870:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201864.708873:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187634688:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201864.708880:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187634816:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788201864.708898:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201864.708909:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201864/real 1788201864] req@ffff8e82894e6300 x1875062187634816/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201864.708921:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187634816/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201864.708929:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201864.708933:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201864.708937:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187634816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201864.708943:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187634944:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788201864.708964:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201864.708975:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201864/real 1788201864] req@ffff8e82894e4a80 x1875062187634944/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201864.708988:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187634944/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201864.708994:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201864.708996:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201864.708998:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187634944:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788201864.709010:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187635072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788201864.709024:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201864.709031:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201864/real 1788201864] req@ffff8e82894e6d80 x1875062187635072/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201864.709038:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187635072/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201864.709043:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201864.709045:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201864.709047:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187635072:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201865.412537:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201866.436477:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201867.460543:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201868.484521:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201869.508499:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201869.828425:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201869.828446:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201869.828450:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201869.828459:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3324 00000100:00080000:0.0:1788201869.828469:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201869.828475:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201869.828507:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187635200/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201869.828531:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201869.828534:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201869.828535:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201869.828538:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3324 00000100:00080000:0.0:1788201869.828541:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201869.828544:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201869.828552:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201869.828565:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187635328/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201869.828566:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187635200:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788201869.828580:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201869.828583:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201869.828584:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201869.828587:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3324 00000100:00080000:0.0:1788201869.828589:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201869.828592:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201869.828595:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201869.828604:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187635456/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201869.828611:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201869.828614:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201869.828615:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201869.828618:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3324 00000100:00080000:0.0:1788201869.828621:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201869.828623:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201869.828627:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788201869.828634:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201869.828639:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187635584/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201869.828651:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201869.828655:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201869.828657:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201869.828661:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3324 00000100:00080000:0.0:1788201869.828666:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201869.828671:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201869.828677:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201869.828682:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201869/real 1788201869] req@ffff8e828793f100 x1875062187635200/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201869.828695:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187635712/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201869.828700:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187635200/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201869.828723:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201869.828728:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201869.828731:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187635200:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201869.828749:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187635328:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201869.828765:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201869.828774:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201869/real 1788201869] req@ffff8e828793ce00 x1875062187635328/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201869.828783:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187635328/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201869.828788:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201869.828790:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201869.828792:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187635328:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201869.828796:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187635456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201869.828806:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201869.828812:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201869/real 1788201869] req@ffff8e828793f800 x1875062187635456/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201869.828824:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187635456/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201869.828829:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201869.828831:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201869.828832:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187635456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201869.828836:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187635584:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201869.828846:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201869.828851:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201869/real 1788201869] req@ffff8e828793e680 x1875062187635584/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201869.828858:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187635584/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201869.828862:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201869.828864:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201869.828866:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187635584:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201869.828870:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187635712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201869.828879:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201869.828884:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201869/real 1788201869] req@ffff8e828793e300 x1875062187635712/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201869.828891:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187635712/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201869.828900:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201869.828902:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201869.828904:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187635712:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201870.532508:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201871.556575:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201872.580588:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201873.604555:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201874.628469:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201874.948445:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201874.948454:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201874.948457:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201874.948465:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3330 00000100:00080000:0.0:1788201874.948475:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201874.948481:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201874.948514:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187635840/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201874.948539:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201874.948544:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201874.948546:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201874.948551:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3330 00000100:00080000:0.0:1788201874.948555:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201874.948559:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201874.948569:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201874.948584:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187635968/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201874.948594:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187635840:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788201874.948596:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201874.948602:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201874.948606:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201874.948609:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3330 00000100:00080000:0.0:1788201874.948612:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201874.948615:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201874.948619:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201874.948627:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187636096/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201874.948634:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201874.948637:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201874.948638:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201874.948642:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3330 00000100:00080000:0.0:1788201874.948644:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201874.948647:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201874.948650:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201874.948656:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187636224/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201874.948662:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788201874.948667:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788201874.948676:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201874.948679:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201874.948685:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3330 00000100:00080000:0.0:1788201874.948699:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201874.948703:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201874.948709:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201874.948722:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187636352/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201874.948728:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201874/real 1788201874] req@ffff8e828793e300 x1875062187635840/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201874.948752:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187635840/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201874.948763:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201874.948768:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201874.948773:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187635840:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201874.948796:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187635968:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201874.948821:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201874.948837:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201874/real 1788201874] req@ffff8e828793e680 x1875062187635968/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201874.948850:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187635968/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201874.948858:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201874.948862:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201874.948866:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187635968:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201874.948872:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187636096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201874.948891:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201874.948901:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201874/real 1788201874] req@ffff8e828793f800 x1875062187636096/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201874.948912:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187636096/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201874.948921:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201874.948923:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201874.948926:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187636096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201874.948932:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187636224:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201874.948949:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201874.948958:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201874/real 1788201874] req@ffff8e828793ce00 x1875062187636224/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201874.948969:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187636224/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201874.948977:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201874.948980:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201874.948984:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187636224:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201874.948989:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187636352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201874.949005:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201874.949014:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201874/real 1788201874] req@ffff8e828793f100 x1875062187636352/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201874.949026:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187636352/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201874.949034:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201874.949037:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201874.949040:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187636352:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201875.652494:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201876.676483:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201877.700485:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201878.724495:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201879.748499:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201880.068448:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201880.068460:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201880.068465:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201880.068476:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3335 00000100:00080000:0.0:1788201880.068488:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201880.068496:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201880.068544:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187636480/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201880.068567:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201880.068571:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201880.068573:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201880.068577:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3335 00000100:00080000:0.0:1788201880.068582:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201880.068586:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201880.068598:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201880.068608:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187636480:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788201880.068611:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187636608/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201880.068621:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201880.068624:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201880.068626:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201880.068630:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3335 00000100:00080000:0.0:1788201880.068633:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201880.068636:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201880.068640:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201880.068648:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187636736/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201880.068657:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201880.068661:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201880.068662:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201880.068666:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3335 00000800:00000100:2.0:1788201880.068668:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788201880.068669:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201880.068673:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201880.068677:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201880.068686:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187636864/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201880.068695:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201880.068698:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201880.068700:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201880.068704:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3335 00000100:00080000:0.0:1788201880.068707:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201880.068712:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201880.068718:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201880.068725:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201880/real 1788201880] req@ffff8e828793c700 x1875062187636480/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201880.068733:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187636992/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201880.068744:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187636480/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201880.068755:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201880.068760:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201880.068764:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187636480:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201880.068787:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187636608:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201880.068806:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201880.068816:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201880/real 1788201880] req@ffff8e828793ca80 x1875062187636608/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201880.068824:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187636608/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201880.068839:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201880.068843:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201880.068846:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187636608:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201880.068852:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187636736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201880.068875:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201880.068887:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201880/real 1788201880] req@ffff8e828793f480 x1875062187636736/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201880.068896:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187636736/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201880.068905:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201880.068908:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201880.068912:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187636736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201880.068918:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187636864:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201880.068940:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201880.068952:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201880/real 1788201880] req@ffff8e828793c000 x1875062187636864/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201880.068961:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187636864/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201880.068966:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201880.068968:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201880.068971:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187636864:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201880.068975:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187636992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201880.068988:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201880.068994:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201880/real 1788201880] req@ffff8e828793dc00 x1875062187636992/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201880.069002:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187636992/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201880.069007:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201880.069008:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201880.069010:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187636992:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201880.772445:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201881.796473:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201882.820571:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201883.844484:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201884.868551:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201885.188404:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201885.188415:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201885.188420:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201885.188429:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3340 00000100:00080000:0.0:1788201885.188440:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201885.188448:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201885.188486:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793df80 x1875062187637120/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201885.188519:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201885.188523:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201885.188526:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201885.188530:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3340 00000100:00080000:0.0:1788201885.188535:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201885.188539:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201885.188550:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201885.188565:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187637248/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201885.188569:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187637120:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788201885.188581:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201885.188584:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201885.188585:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201885.188588:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3340 00000100:00080000:0.0:1788201885.188591:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201885.188594:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201885.188598:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201885.188606:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187637376/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201885.188617:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201885.188620:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201885.188621:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201885.188623:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3340 00000100:00080000:0.0:1788201885.188626:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201885.188629:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201885.188632:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201885.188640:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187637504/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201885.188643:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788201885.188651:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201885.188654:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201885.188656:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201885.188660:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3340 00000100:00080000:0.0:1788201885.188664:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201885.188666:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201885.188669:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201885.188676:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187637632/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201885.188692:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201885/real 1788201885] req@ffff8e828793df80 x1875062187637120/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201885.188709:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793df80 x1875062187637120/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201885.188717:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201885.188720:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201885.188723:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187637120:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201885.188740:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187637248:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201885.188756:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201885.188765:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201885/real 1788201885] req@ffff8e828793d500 x1875062187637248/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201885.188774:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187637248/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201885.188779:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201885.188781:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201885.188784:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187637248:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201885.188787:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187637376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201885.188798:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201885.188804:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201885/real 1788201885] req@ffff8e828793d880 x1875062187637376/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201885.188811:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187637376/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201885.188816:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201885.188818:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201885.188819:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187637376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201885.188823:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187637504:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201885.188833:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201885.188838:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201885/real 1788201885] req@ffff8e828793c380 x1875062187637504/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201885.188845:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187637504/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201885.188850:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201885.188852:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201885.188854:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187637504:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201885.188858:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187637632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201885.188867:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201885.188872:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201885/real 1788201885] req@ffff8e828793fb80 x1875062187637632/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201885.188878:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187637632/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201885.188883:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201885.188885:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201885.188887:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187637632:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201885.892511:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201886.916503:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201887.940470:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201888.964549:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201889.988506:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201890.308543:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201890.308552:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201890.308558:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201890.308571:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3345 00000100:00080000:0.0:1788201890.308584:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201890.308593:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201890.308634:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187637760/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201890.308664:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201890.308668:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201890.308670:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201890.308674:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3345 00000100:00080000:0.0:1788201890.308676:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201890.308680:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201890.308690:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201890.308706:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187637888/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201890.308716:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201890.308722:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788201890.308724:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187637760:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788201890.308725:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201890.308733:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3345 00000100:00080000:0.0:1788201890.308740:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201890.308746:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201890.308750:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201890.308767:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187638016/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201890.308786:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201890.308790:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201890.308792:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201890.308797:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3345 00000100:00080000:0.0:1788201890.308803:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788201890.308811:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788201890.308819:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201890.308828:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201890.308844:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187638144/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201890.308852:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201890.308855:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201890.308857:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201890.308860:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3345 00000100:00080000:0.0:1788201890.308863:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201890.308866:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201890.308869:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201890.308878:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793df80 x1875062187638272/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201890.308952:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201890/real 1788201890] req@ffff8e828793fb80 x1875062187637760/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201890.308976:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187637760/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201890.308987:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201890.308992:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201890.308996:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187637760:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788201890.309036:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187637888:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201890.309063:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201890.309075:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201890/real 1788201890] req@ffff8e828793c380 x1875062187637888/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201890.309100:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187637888/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201890.309109:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201890.309113:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201890.309117:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187637888:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201890.309124:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187638016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201890.309148:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201890.309161:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201890/real 1788201890] req@ffff8e828793d880 x1875062187638016/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201890.309171:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187638016/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201890.309176:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201890.309179:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201890.309181:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187638016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201890.309185:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187638144:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201890.309201:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201890.309208:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201890/real 1788201890] req@ffff8e828793d500 x1875062187638144/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201890.309219:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187638144/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201890.309234:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201890.309237:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201890.309241:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187638144:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201890.309406:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187638272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201890.309431:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201890.309446:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201890/real 1788201890] req@ffff8e828793df80 x1875062187638272/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201890.309518:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793df80 x1875062187638272/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201890.309530:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201890.309535:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201890.309541:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187638272:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201891.012571:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201892.036516:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201893.060501:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201894.084477:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201895.108485:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788201895.428394:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201895.428403:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788201895.428407:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201895.428415:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3350 00000100:00080000:3.0:1788201895.428424:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201895.428430:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788201895.428461:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187638400/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201895.428484:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201895.428487:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788201895.428488:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201895.428491:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3350 00000100:00080000:3.0:1788201895.428494:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201895.428497:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201895.428505:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201895.428514:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187638528/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201895.428523:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187638400:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788201895.428525:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201895.428531:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788201895.428534:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201895.428538:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3350 00000100:00080000:3.0:1788201895.428540:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201895.428542:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201895.428545:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201895.428555:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187638656/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201895.428561:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201895.428563:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788201895.428565:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201895.428568:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3350 00000100:00080000:3.0:1788201895.428581:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201895.428584:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201895.428588:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788201895.428599:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201895.428601:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187638784/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201895.428623:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201895.428628:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788201895.428630:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788201895.428634:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3350 00000100:00080000:3.0:1788201895.428638:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788201895.428644:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788201895.428649:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201895.428654:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201895/real 1788201895] req@ffff8e8289539880 x1875062187638400/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201895.428667:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187638912/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201895.428674:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187638400/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201895.428692:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201895.428695:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201895.428699:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187638400:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201895.428719:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187638528:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201895.428742:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201895.428756:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201895/real 1788201895] req@ffff8e828953a680 x1875062187638528/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201895.428771:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187638528/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201895.428780:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201895.428783:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201895.428786:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187638528:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201895.428790:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187638656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201895.428807:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201895.428815:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201895/real 1788201895] req@ffff8e8289538a80 x1875062187638656/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201895.428822:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187638656/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201895.428828:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201895.428830:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201895.428832:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187638656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201895.428835:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187638784:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201895.428846:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201895.428852:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201895/real 1788201895] req@ffff8e8289539500 x1875062187638784/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201895.428859:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187638784/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201895.428864:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201895.428866:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201895.428868:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187638784:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201895.428871:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187638912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201895.428883:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201895.428892:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201895/real 1788201895] req@ffff8e8289539f80 x1875062187638912/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201895.428904:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187638912/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201895.428913:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201895.428916:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201895.428920:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187638912:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201896.132462:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201897.156470:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201898.180452:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201899.204520:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201900.228464:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201900.548438:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201900.548449:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201900.548453:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201900.548474:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3355 00000100:00080000:2.0:1788201900.548488:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201900.548496:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788201900.548541:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187639040/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201900.548580:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201900.548585:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201900.548587:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201900.548592:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3355 00000100:00080000:2.0:1788201900.548596:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201900.548600:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201900.548610:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201900.548626:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187639040:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201900.548628:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187639168/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201900.548644:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201900.548649:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201900.548651:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201900.548656:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3355 00000100:00080000:2.0:1788201900.548660:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201900.548665:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201900.548671:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201900.548684:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187639296/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201900.548695:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788201900.548701:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788201900.548701:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788201900.548706:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201900.548712:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3355 00000100:00080000:2.0:1788201900.548717:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201900.548722:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201900.548727:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201900.548739:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187639424/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201900.548748:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201900/real 1788201900] req@ffff8e82894e5f80 x1875062187639040/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201900.548750:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201900.548757:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788201900.548761:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201900.548768:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3355 00000100:00080000:2.0:1788201900.548772:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201900.548777:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201900.548778:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187639040/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788201900.548782:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788201900.548792:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201900.548796:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201900.548799:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187639552/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201900.548803:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187639040:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788201900.548832:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187639168:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788201900.548851:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201900.548859:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201900/real 1788201900] req@ffff8e82894e5c00 x1875062187639168/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201900.548868:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187639168/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201900.548874:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201900.548877:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201900.548880:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187639168:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201900.548887:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187639296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788201900.548919:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201900.548933:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201900/real 1788201900] req@ffff8e82894e4380 x1875062187639296/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201900.548946:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187639296/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201900.548956:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201900.548958:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201900.548960:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187639296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201900.548965:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187639424:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788201900.548980:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201900.548986:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201900/real 1788201900] req@ffff8e82894e7100 x1875062187639424/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201900.548994:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187639424/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201900.548999:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201900.549001:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201900.549003:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187639424:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788201900.549006:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187639552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788201900.549016:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201900.549021:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201900/real 1788201900] req@ffff8e82894e6d80 x1875062187639552/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201900.549028:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187639552/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201900.549033:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201900.549035:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201900.549038:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187639552:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201901.252497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201902.276535:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201903.300484:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201904.324489:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201905.348471:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201905.668428:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201905.668437:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201905.668440:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201905.668448:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3360 00000100:00080000:1.0:1788201905.668458:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201905.668463:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788201905.668496:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187639680/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201905.668518:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201905.668521:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201905.668522:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201905.668525:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3360 00000100:00080000:1.0:1788201905.668527:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201905.668530:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201905.668537:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201905.668547:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187639808/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201905.668553:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201905.668556:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788201905.668557:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201905.668559:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3360 00000100:00080000:1.0:1788201905.668560:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201905.668562:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201905.668565:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201905.668571:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187639936/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201905.668579:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201905.668581:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201905.668583:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201905.668586:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3360 00000100:00080000:1.0:1788201905.668587:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201905.668590:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201905.668593:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788201905.668598:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187640064/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201905.668605:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201905.668607:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201905.668608:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201905.668610:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3360 00000100:00080000:1.0:1788201905.668613:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201905.668622:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201905.668623:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187639680:192.168.201.121@tcp:250:kworker.0 00000008:00100000:1.0:1788201905.668629:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788201905.668641:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187640192/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201905.668746:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187639808:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201905.668776:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187639936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201905.668802:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187640064:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788201905.668839:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187640192:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201906.372480:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201907.396457:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201908.420580:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788201908.740555:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:1.0:1788201908.740591:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788201908.740606:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788201908.740645:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788201908.740656:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788201908.740665:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788201908.740673:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:0.0:1788201908.740683:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201905/real 1788201908] req@ffff8e828d722300 x1875062187640064/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201908.740708:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187640064/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201908.740720:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201908.740725:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201908.740730:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187640064:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788201908.740748:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201905/real 1788201908] req@ffff8e828d720380 x1875062187640192/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201908.740794:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187640192/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201908.740805:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201908.740808:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201908.740812:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187640192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788201908.740820:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201905/real 1788201908] req@ffff8e828d720a80 x1875062187639680/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201908.740832:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187639680/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201908.740839:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201908.740843:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201908.740846:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187639680:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788201908.740854:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201905/real 1788201908] req@ffff8e828d721180 x1875062187639808/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201908.740865:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187639808/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201908.740873:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201908.740876:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201908.740879:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187639808:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201908.740886:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201905/real 1788201908] req@ffff8e828d722d80 x1875062187639936/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201908.740896:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187639936/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201908.740903:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201908.740907:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201908.740910:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187639936:192.168.201.121@tcp:38:kworker.0 00010000:00010000:0.0:1788201909.444542:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201910.468569:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201910.788444:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201910.788453:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201910.788457:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201910.788466:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3365 00000100:00080000:1.0:1788201910.788475:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201910.788480:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788201910.788508:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187640320/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201910.788531:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201910.788534:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201910.788536:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201910.788539:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3365 00000100:00080000:1.0:1788201910.788541:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201910.788543:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201910.788556:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201910.788570:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187640448/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201910.788581:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201910.788584:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788201910.788586:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201910.788591:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3365 00000100:00100000:0.0:1788201910.788597:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187640320:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788201910.788597:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201910.788606:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201910.788612:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201910.788626:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187640576/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201910.788638:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201910.788642:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201910.788644:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201910.788649:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3365 00000100:00080000:1.0:1788201910.788652:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201910.788655:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:0.0:1788201910.788660:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:1.0:1788201910.788660:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788201910.788674:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187640704/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201910.788685:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201910.788688:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201910.788690:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201910.788695:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3365 00000100:00080000:1.0:1788201910.788700:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201910.788704:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201910.788709:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201910/real 1788201910] req@ffff8e828d720e00 x1875062187640320/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788201910.788712:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201910.788748:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187640320/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201910.788760:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187640832/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201910.788762:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201910.788772:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201910.788780:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187640320:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788201910.788796:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187640448:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788201910.788823:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201910.788838:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201910/real 1788201910] req@ffff8e828d720000 x1875062187640448/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201910.788851:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187640448/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201910.788861:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201910.788865:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201910.788869:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187640448:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201910.788894:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187640576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788201910.788915:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201910.788945:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201910/real 1788201910] req@ffff8e828d721500 x1875062187640576/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201910.788958:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187640576/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201910.788966:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201910.788970:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201910.788973:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187640576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201910.788979:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187640704:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788201910.788998:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201910.789008:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201910/real 1788201910] req@ffff8e828d723800 x1875062187640704/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201910.789019:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187640704/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201910.789027:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201910.789030:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201910.789034:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187640704:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788201910.789040:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187640832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788201910.789057:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201910.789067:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201910/real 1788201910] req@ffff8e828d722680 x1875062187640832/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201910.789078:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187640832/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201910.789086:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201910.789089:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201910.789092:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187640832:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201911.492517:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201912.516505:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201913.540521:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201914.564461:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201915.588460:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201915.908458:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201915.908473:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201915.908478:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201915.908494:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3370 00000100:00080000:0.0:1788201915.908508:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201915.908518:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201915.908571:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187640960/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201915.908606:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201915.908610:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201915.908613:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201915.908619:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3370 00000100:00080000:0.0:1788201915.908624:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201915.908630:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201915.908653:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201915.908670:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187641088/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201915.908682:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201915.908686:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201915.908689:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201915.908694:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3370 00000100:00080000:0.0:1788201915.908698:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201915.908705:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201915.908711:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201915.908716:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187640960:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788201915.908722:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187641216/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201915.908731:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201915.908734:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201915.908736:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201915.908739:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3370 00000100:00080000:0.0:1788201915.908741:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201915.908744:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201915.908748:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201915.908755:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187641344/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201915.908767:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201915.908770:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201915.908771:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201915.908773:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3370 00000100:00080000:0.0:1788201915.908776:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201915.908778:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201915.908781:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201915.908788:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187641472/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201915.908799:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201915.908853:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201915/real 1788201915] req@ffff8e828793dc00 x1875062187640960/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201915.908871:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187640960/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201915.908881:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201915.908885:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201915.908888:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187640960:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201915.908895:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187641088:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201915.908918:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201915.908932:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201915/real 1788201915] req@ffff8e828793c000 x1875062187641088/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201915.908946:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187641088/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201915.908956:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201915.908959:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201915.908963:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187641088:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201915.908993:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187641216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201915.909017:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201915.909028:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201915/real 1788201915] req@ffff8e828793f480 x1875062187641216/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201915.909036:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187641216/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201915.909042:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201915.909044:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201915.909046:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187641216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201915.909050:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187641344:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201915.909064:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201915.909070:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201915/real 1788201915] req@ffff8e828793ca80 x1875062187641344/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201915.909083:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187641344/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201915.909091:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201915.909094:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201915.909097:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187641344:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201915.909104:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187641472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201915.909124:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201915.909135:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201915/real 1788201915] req@ffff8e828793c700 x1875062187641472/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201915.909143:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187641472/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201915.909149:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201915.909151:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201915.909153:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187641472:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201916.612498:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201917.636443:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201918.660535:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201919.684499:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201920.708450:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201921.028391:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201921.028402:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201921.028415:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201921.028428:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3376 00000100:00080000:0.0:1788201921.028438:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201921.028445:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201921.028477:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187641600/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201921.028500:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201921.028502:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201921.028507:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201921.028512:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3376 00000100:00080000:0.0:1788201921.028514:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201921.028517:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201921.028524:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201921.028533:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187641728/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201921.028540:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201921.028543:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201921.028544:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201921.028560:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3376 00000100:00080000:0.0:1788201921.028563:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201921.028565:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788201921.028571:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187641600:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788201921.028572:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201921.028585:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187641856/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201921.028593:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201921.028595:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201921.028597:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201921.028599:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3376 00000100:00080000:0.0:1788201921.028602:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201921.028605:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201921.028608:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201921.028617:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187641984/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201921.028624:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201921.028626:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201921.028628:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201921.028631:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3376 00000800:00000100:2.0:1788201921.028635:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788201921.028637:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201921.028642:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201921.028646:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201921.028654:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187642112/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201921.028691:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201921/real 1788201921] req@ffff8e828793f100 x1875062187641600/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201921.028706:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187641600/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201921.028718:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201921.028722:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201921.028724:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187641600:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201921.028730:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187641728:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201921.028746:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201921.028755:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201921/real 1788201921] req@ffff8e828793ce00 x1875062187641728/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201921.028763:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187641728/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201921.028768:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201921.028771:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201921.028773:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187641728:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201921.028791:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187641856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201921.028803:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201921.028809:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201921/real 1788201921] req@ffff8e828793f800 x1875062187641856/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201921.028817:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187641856/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201921.028822:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201921.028824:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201921.028826:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187641856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201921.028830:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187641984:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201921.028841:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201921.028847:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201921/real 1788201921] req@ffff8e828793e680 x1875062187641984/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201921.028854:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187641984/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201921.028860:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201921.028861:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201921.028863:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187641984:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201921.028867:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187642112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201921.028876:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201921.028881:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201921/real 1788201921] req@ffff8e828793e300 x1875062187642112/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201921.028888:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187642112/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201921.028892:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201921.028894:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201921.028896:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187642112:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201921.732472:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201922.756486:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201923.780491:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201924.804470:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201925.828482:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201926.148411:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201926.148420:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201926.148425:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201926.148436:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3381 00000100:00080000:0.0:1788201926.148448:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201926.148455:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201926.148507:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187642240/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201926.148537:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201926.148541:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201926.148544:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201926.148552:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3381 00000100:00080000:0.0:1788201926.148558:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201926.148565:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788201926.148566:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187642240:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788201926.148581:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201926.148598:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187642368/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201926.148610:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201926.148614:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201926.148616:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201926.148620:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3381 00000800:00000100:3.0:1788201926.148634:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788201926.148636:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201926.148647:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201926.148654:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201926.148668:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187642496/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201926.148678:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201926/real 1788201926] req@ffff8e828793e300 x1875062187642240/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201926.148680:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201926.148689:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201926.148692:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201926.148698:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3381 00000100:00080000:0.0:1788201926.148703:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201926.148708:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201926.148713:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201926.148723:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187642624/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201926.148733:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201926.148736:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201926.148738:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201926.148742:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3381 00000100:00080000:0.0:1788201926.148746:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201926.148749:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201926.148755:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201926.148770:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187642752/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201926.148779:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187642240/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201926.148790:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201926.148794:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201926.148798:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187642240:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788201926.148817:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187642368:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201926.148835:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201926.148844:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201926/real 1788201926] req@ffff8e828793e680 x1875062187642368/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201926.148852:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187642368/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201926.148858:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201926.148860:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201926.148862:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187642368:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201926.148866:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187642496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201926.148876:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201926.148882:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201926/real 1788201926] req@ffff8e828793f800 x1875062187642496/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201926.148889:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187642496/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201926.148894:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201926.148896:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201926.148898:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187642496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201926.148901:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187642624:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201926.148912:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201926.149020:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201926/real 1788201926] req@ffff8e828793ce00 x1875062187642624/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201926.149038:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187642624/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201926.149045:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201926.149053:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201926.149056:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187642624:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201926.149061:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187642752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201926.149079:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201926.149128:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201926/real 1788201926] req@ffff8e828793f100 x1875062187642752/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201926.149137:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187642752/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201926.149143:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201926.149145:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201926.149147:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187642752:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201926.852517:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201927.876470:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201928.900459:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201929.924556:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201930.948466:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201931.268495:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201931.268506:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201931.268511:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201931.268521:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3386 00000100:00080000:1.0:1788201931.268532:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201931.268539:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788201931.268602:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187642880/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201931.268636:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201931.268640:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201931.268642:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201931.268647:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3386 00000100:00080000:1.0:1788201931.268651:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201931.268656:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201931.268666:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201931.268698:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187643008/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201931.268704:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187642880:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788201931.268716:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201931.268720:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788201931.268722:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201931.268726:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3386 00000100:00080000:1.0:1788201931.268729:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201931.268733:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201931.268738:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201931.268750:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187643136/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201931.268758:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201931.268779:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201931.268781:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201931.268786:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3386 00000100:00080000:1.0:1788201931.268791:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201931.268795:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:3.0:1788201931.268805:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:1.0:1788201931.268808:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788201931.268825:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187643264/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201931.268836:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201931.268840:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201931.268842:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201931.268848:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3386 00000100:00080000:1.0:1788201931.268852:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201931.268857:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788201931.268864:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201931/real 1788201931] req@ffff8e828d722680 x1875062187642880/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788201931.268867:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788201931.268902:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187643392/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201931.268911:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187642880/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201931.268929:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201931.268934:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201931.268939:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187642880:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788201931.268962:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187643008:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201931.268982:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201931.268993:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201931/real 1788201931] req@ffff8e828d723800 x1875062187643008/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201931.269003:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187643008/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201931.269010:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201931.269013:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201931.269015:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187643008:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201931.269020:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187643136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201931.269033:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201931.269040:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201931/real 1788201931] req@ffff8e828d721500 x1875062187643136/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201931.269050:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187643136/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201931.269056:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201931.269058:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201931.269061:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187643136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201931.269066:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187643264:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201931.269079:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201931.269086:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201931/real 1788201931] req@ffff8e828d720000 x1875062187643264/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201931.269095:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187643264/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201931.269101:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201931.269103:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201931.269106:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187643264:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201931.269110:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187643392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201931.269122:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201931.269129:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201931/real 1788201931] req@ffff8e828d720e00 x1875062187643392/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201931.269137:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187643392/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201931.269143:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201931.269146:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201931.269148:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187643392:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201931.972592:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201932.996507:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201934.020483:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201935.044499:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201936.068526:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201936.388388:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201936.388395:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201936.388399:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201936.388407:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3391 00000100:00080000:1.0:1788201936.388417:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201936.388423:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788201936.388454:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187643520/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201936.388478:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201936.388480:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201936.388482:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201936.388485:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3391 00000100:00080000:1.0:1788201936.388487:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201936.388489:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201936.388496:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201936.388504:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187643648/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201936.388512:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201936.388515:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788201936.388519:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788201936.388520:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187643520:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788201936.388524:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3391 00000100:00080000:1.0:1788201936.388529:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201936.388531:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201936.388535:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201936.388542:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187643776/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201936.388549:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201936.388551:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201936.388553:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201936.388555:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3391 00000100:00080000:1.0:1788201936.388557:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201936.388559:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201936.388562:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788201936.388570:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187643904/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201936.388576:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201936.388578:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201936.388579:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201936.388582:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3391 00000100:00080000:1.0:1788201936.388585:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201936.388590:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:3.0:1788201936.388590:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:1.0:1788201936.388597:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788201936.388605:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187644032/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201936.388635:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201936/real 1788201936] req@ffff8e828d722d80 x1875062187643520/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201936.388650:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187643520/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201936.388658:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201936.388661:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201936.388664:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187643520:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788201936.388683:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187643648:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201936.388700:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201936.388708:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201936/real 1788201936] req@ffff8e828d721180 x1875062187643648/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201936.388716:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187643648/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201936.388722:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201936.388724:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201936.388726:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187643648:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201936.388730:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187643776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201936.388740:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201936.388747:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201936/real 1788201936] req@ffff8e828d720a80 x1875062187643776/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201936.388754:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187643776/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201936.388763:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201936.388765:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201936.388767:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187643776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201936.388771:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187643904:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201936.388781:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201936.388787:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201936/real 1788201936] req@ffff8e828d720380 x1875062187643904/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201936.388793:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187643904/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201936.388798:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201936.388800:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201936.388802:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187643904:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201936.388805:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187644032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201936.388814:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201936.388820:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201936/real 1788201936] req@ffff8e828d722300 x1875062187644032/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201936.388826:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187644032/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201936.388831:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201936.388833:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201936.388835:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187644032:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201937.092452:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201938.116498:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201939.140461:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201940.164569:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201941.188505:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201941.508416:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201941.508424:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201941.508428:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201941.508436:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3396 00000100:00080000:0.0:1788201941.508446:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201941.508464:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201941.508496:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187644160/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201941.508520:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201941.508524:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201941.508526:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201941.508530:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3396 00000100:00080000:0.0:1788201941.508535:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201941.508539:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201941.508549:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201941.508556:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187644160:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788201941.508567:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187644288/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201941.508577:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201941.508580:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201941.508582:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201941.508584:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3396 00000100:00080000:0.0:1788201941.508587:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201941.508590:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201941.508594:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201941.508602:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187644416/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201941.508608:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201941.508610:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201941.508612:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201941.508615:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3396 00000100:00080000:0.0:1788201941.508619:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201941.508622:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201941.508626:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788201941.508632:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201941.508634:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187644544/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201941.508642:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201941.508645:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201941.508646:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201941.508649:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3396 00000100:00080000:0.0:1788201941.508658:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201941.508661:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201941.508664:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201941.508673:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187644672/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201941.508681:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201941/real 1788201941] req@ffff8e828793c700 x1875062187644160/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201941.508698:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187644160/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201941.508706:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201941.508709:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201941.508712:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187644160:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788201941.508730:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187644288:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201941.508748:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201941.508757:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201941/real 1788201941] req@ffff8e828793ca80 x1875062187644288/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201941.508765:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187644288/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201941.508770:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201941.508772:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201941.508774:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187644288:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201941.508778:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187644416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201941.508788:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201941.508795:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201941/real 1788201941] req@ffff8e828793f480 x1875062187644416/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201941.508803:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187644416/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201941.508808:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201941.508817:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201941.508819:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187644416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201941.508823:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187644544:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201941.508841:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201941.508852:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201941/real 1788201941] req@ffff8e828793c000 x1875062187644544/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201941.508865:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187644544/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201941.508873:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201941.508877:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201941.508880:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187644544:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201941.508887:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187644672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201941.508904:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201941.508912:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201941/real 1788201941] req@ffff8e828793dc00 x1875062187644672/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201941.508919:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187644672/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201941.508933:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201941.508935:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201941.508938:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187644672:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201942.212429:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201943.236505:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201944.260549:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201945.284503:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201946.308523:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201946.628467:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201946.628480:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201946.628484:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201946.628494:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3401 00000100:00080000:0.0:1788201946.628505:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201946.628512:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201946.628550:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793df80 x1875062187644800/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201946.628586:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201946.628590:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201946.628591:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201946.628595:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3401 00000100:00080000:0.0:1788201946.628598:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201946.628603:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201946.628614:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201946.628620:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187644800:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788201946.628635:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187644928/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201946.628650:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201946.628656:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201946.628659:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201946.628663:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3401 00000100:00080000:0.0:1788201946.628670:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201946.628673:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201946.628679:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201946.628703:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187645056/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788201946.628710:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788201946.628724:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201946.628728:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201946.628731:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201946.628737:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3401 00000100:00080000:0.0:1788201946.628743:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201946.628749:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201946.628755:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201946.628768:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187645184/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201946.628777:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788201946.628779:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201946/real 1788201946] req@ffff8e828793df80 x1875062187644800/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201946.628781:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201946.628784:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201946.628794:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3401 00000100:00080000:0.0:1788201946.628801:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201946.628806:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201946.628811:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788201946.628814:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793df80 x1875062187644800/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201946.628824:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187645312/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788201946.628828:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201946.628838:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201946.628846:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187644800:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788201946.628878:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187644928:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201946.628901:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201946.628912:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201946/real 1788201946] req@ffff8e828793d500 x1875062187644928/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201946.628922:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187644928/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201946.628929:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201946.628931:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201946.628934:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187644928:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201946.628939:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187645056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201946.628952:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201946.628959:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201946/real 1788201946] req@ffff8e828793d880 x1875062187645056/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201946.628968:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187645056/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201946.628974:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201946.628976:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201946.628978:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187645056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201946.628983:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187645184:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201946.628995:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201946.629002:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201946/real 1788201946] req@ffff8e828793c380 x1875062187645184/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201946.629010:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187645184/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201946.629016:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201946.629019:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201946.629021:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187645184:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201946.629025:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187645312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201946.629036:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201946.629042:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201946/real 1788201946] req@ffff8e828793fb80 x1875062187645312/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201946.629051:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187645312/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201946.629057:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201946.629059:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201946.629061:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187645312:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201947.332446:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201948.356466:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201949.380521:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201950.404458:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201951.428514:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201951.748421:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201951.748429:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201951.748433:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201951.748441:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3406 00000100:00080000:1.0:1788201951.748451:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201951.748458:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788201951.748490:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187645440/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201951.748512:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201951.748515:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201951.748517:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201951.748523:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3406 00000100:00080000:1.0:1788201951.748527:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201951.748533:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201951.748543:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201951.748548:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187645440:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788201951.748562:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187645568/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201951.748576:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201951.748581:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788201951.748582:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201951.748586:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3406 00000100:00080000:1.0:1788201951.748590:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201951.748593:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201951.748600:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788201951.748608:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788201951.748617:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187645696/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201951.748657:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201951.748663:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788201951.748664:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201951/real 1788201951] req@ffff8e828d722300 x1875062187645440/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201951.748666:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201951.748673:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3406 00000100:00080000:1.0:1788201951.748678:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201951.748685:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201951.748690:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788201951.748698:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187645440/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201951.748702:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187645824/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201951.748747:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788201951.748749:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201951.748754:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:1.0:1788201951.748757:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788201951.748758:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187645440:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788201951.748760:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201951.748769:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3406 00000100:00080000:1.0:1788201951.748773:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201951.748776:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201951.748780:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788201951.748804:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187645568:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788201951.748808:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187645952/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788201951.748831:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201951.748843:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201951/real 1788201951] req@ffff8e828d720380 x1875062187645568/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201951.748851:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187645568/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201951.748861:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201951.748864:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201951.748866:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187645568:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201951.748870:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187645696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201951.748880:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201951.748886:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201951/real 1788201951] req@ffff8e828d720a80 x1875062187645696/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201951.748893:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187645696/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201951.748898:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201951.748900:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201951.748902:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187645696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201951.748905:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187645824:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201951.748916:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201951.748921:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201951/real 1788201951] req@ffff8e828d721180 x1875062187645824/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201951.748929:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187645824/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201951.748933:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201951.748935:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201951.748938:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187645824:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201951.748949:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187645952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201951.748959:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201951.748965:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201951/real 1788201951] req@ffff8e828d722d80 x1875062187645952/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201951.748972:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187645952/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201951.748977:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201951.748979:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201951.748981:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187645952:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201952.452533:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201953.476498:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201954.500470:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201955.524488:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201956.548514:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201956.868421:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201956.868432:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201956.868437:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201956.868460:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3411 00000100:00080000:1.0:1788201956.868473:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201956.868480:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788201956.868523:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187646080/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201956.868550:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201956.868553:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201956.868554:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201956.868557:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3411 00000100:00080000:1.0:1788201956.868560:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201956.868562:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201956.868572:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201956.868581:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187646208/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201956.868596:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201956.868602:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788201956.868607:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201956.868611:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3411 00000100:00100000:3.0:1788201956.868613:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187646080:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788201956.868615:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201956.868627:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201956.868634:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201956.868651:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187646336/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201956.868658:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201956.868661:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201956.868662:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201956.868665:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3411 00000100:00080000:1.0:1788201956.868667:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201956.868670:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201956.868675:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788201956.868683:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187646464/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201956.868690:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201956.868693:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201956.868694:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201956.868701:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3411 00000800:00000100:3.0:1788201956.868702:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788201956.868705:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201956.868709:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201956.868713:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788201956.868732:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187646592/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201956.868778:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201956/real 1788201956] req@ffff8e828d720e00 x1875062187646080/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201956.868796:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187646080/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201956.868806:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201956.868810:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201956.868813:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187646080:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788201956.868852:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187646208:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201956.868871:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201956.868881:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201956/real 1788201956] req@ffff8e828d720000 x1875062187646208/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201956.868890:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187646208/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201956.868895:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201956.868897:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201956.868900:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187646208:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201956.868904:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187646336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201956.868914:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201956.868920:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201956/real 1788201956] req@ffff8e828d721500 x1875062187646336/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201956.868944:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187646336/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201956.868955:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201956.868958:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201956.868977:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187646336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201956.868983:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187646464:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201956.869001:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201956.869009:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201956/real 1788201956] req@ffff8e828d723800 x1875062187646464/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201956.869018:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187646464/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201956.869024:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201956.869026:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201956.869029:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187646464:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201956.869049:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187646592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201956.869061:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201956.869067:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201956/real 1788201956] req@ffff8e828d722680 x1875062187646592/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201956.869074:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187646592/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201956.869079:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201956.869081:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201956.869082:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187646592:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201957.572547:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201958.596466:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201959.620530:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201960.644557:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201961.668544:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201961.988440:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201961.988455:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201961.988461:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201961.988477:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3417 00000100:00080000:2.0:1788201961.988492:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201961.988503:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788201961.988552:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187646720/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201961.988591:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201961.988595:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201961.988598:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201961.988606:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3417 00000100:00080000:2.0:1788201961.988610:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201961.988617:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201961.988629:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201961.988647:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187646848/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201961.988660:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201961.988665:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201961.988667:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201961.988673:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3417 00000100:00080000:2.0:1788201961.988679:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201961.988686:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201961.988704:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201961.988705:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187646720:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201961.988723:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187646976/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201961.988749:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201961.988753:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788201961.988756:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201961.988760:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3417 00000100:00080000:2.0:1788201961.988764:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201961.988769:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201961.988775:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788201961.988781:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201961.988791:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187647104/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201961.988803:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201961.988807:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788201961.988810:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201961.988814:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3417 00000100:00080000:2.0:1788201961.988818:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201961.988822:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201961.988828:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201961.988846:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201961/real 1788201961] req@ffff8e82894e4a80 x1875062187646720/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201961.988856:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187647232/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788201961.988876:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187646720/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201961.988891:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201961.988897:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201961.988901:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187646720:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788201961.988910:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187646848:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788201961.988939:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201961.988953:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201961/real 1788201961] req@ffff8e82894e6300 x1875062187646848/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201961.988964:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187646848/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201961.988970:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201961.988973:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201961.988976:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187646848:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201961.989009:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187646976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788201961.989042:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201961.989057:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201961/real 1788201961] req@ffff8e82894e7800 x1875062187646976/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201961.989080:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187646976/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201961.989089:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201961.989094:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201961.989099:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187646976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201961.989106:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187647104:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788201961.989127:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201961.989140:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201961/real 1788201961] req@ffff8e82894e4e00 x1875062187647104/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201961.989161:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187647104/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201961.989167:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201961.989170:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201961.989173:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187647104:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788201961.989178:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187647232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788201961.989194:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788201961.989202:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201961/real 1788201961] req@ffff8e82894e7480 x1875062187647232/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201961.989209:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187647232/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201961.989219:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788201961.989221:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201961.989223:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187647232:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201962.692497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201963.716522:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201964.740516:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201965.764518:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201966.788493:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201967.108432:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201967.108440:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201967.108444:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201967.108452:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3422 00000100:00080000:0.0:1788201967.108461:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201967.108466:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201967.108513:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187647360/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201967.108546:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201967.108549:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201967.108551:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201967.108557:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3422 00000100:00080000:0.0:1788201967.108561:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201967.108564:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201967.108573:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788201967.108575:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187647360:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788201967.108586:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187647488/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201967.108599:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201967.108602:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201967.108604:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201967.108607:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3422 00000100:00080000:0.0:1788201967.108610:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201967.108612:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201967.108615:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201967.108623:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187647616/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201967.108630:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201967.108632:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201967.108634:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201967.108636:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3422 00000100:00080000:0.0:1788201967.108639:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201967.108642:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201967.108646:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201967.108653:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187647744/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788201967.108660:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788201967.108664:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201967.108668:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201967.108669:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201967.108673:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3422 00000100:00080000:0.0:1788201967.108676:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201967.108679:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201967.108682:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201967.108688:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793df80 x1875062187647872/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201967.108710:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201967/real 1788201967] req@ffff8e828793fb80 x1875062187647360/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201967.108729:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187647360/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201967.108741:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201967.108745:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201967.108764:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187647360:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788201967.108784:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187647488:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201967.108803:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201967.108813:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201967/real 1788201967] req@ffff8e828793c380 x1875062187647488/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201967.108822:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187647488/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201967.108828:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201967.108830:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201967.108833:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187647488:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201967.108836:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187647616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201967.108847:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201967.108854:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201967/real 1788201967] req@ffff8e828793d880 x1875062187647616/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201967.108861:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187647616/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201967.108866:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201967.108868:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201967.108870:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187647616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201967.108874:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187647744:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201967.108884:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201967.108890:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201967/real 1788201967] req@ffff8e828793d500 x1875062187647744/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201967.108897:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187647744/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201967.108902:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201967.108904:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201967.108906:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187647744:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201967.108909:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187647872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201967.108919:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201967.108924:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201967/real 1788201967] req@ffff8e828793df80 x1875062187647872/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201967.108931:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793df80 x1875062187647872/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201967.108937:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201967.108938:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201967.108940:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187647872:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201967.812474:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201968.836457:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201969.860452:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201970.884540:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201971.908452:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201972.228392:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201972.228401:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201972.228406:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201972.228415:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3427 00000100:00080000:0.0:1788201972.228427:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201972.228434:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201972.228474:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187648000/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201972.228507:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201972.228512:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201972.228515:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201972.228519:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3427 00000100:00080000:0.0:1788201972.228524:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201972.228528:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201972.228537:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201972.228552:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187648128/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201972.228578:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788201972.228580:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187648000:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788201972.228585:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201972.228589:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201972.228594:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3427 00000100:00080000:0.0:1788201972.228599:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201972.228604:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201972.228610:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201972.228626:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187648256/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201972.228636:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201972.228639:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201972.228642:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201972.228646:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3427 00000100:00080000:0.0:1788201972.228650:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201972.228654:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201972.228671:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201972.228686:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187648384/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201972.228695:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187648128:192.168.201.121@tcp:38:kworker.0 00000100:00080000:0.0:1788201972.228701:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201972.228706:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201972.228708:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201972.228714:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3427 00000100:00080000:0.0:1788201972.228720:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201972.228724:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788201972.228727:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187648256:192.168.201.121@tcp:38:kworker.0 00000008:00100000:0.0:1788201972.228732:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788201972.228746:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187648512/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201972.228751:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187648384:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788201972.228778:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187648512:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201972.932544:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201973.956508:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201974.980517:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788201975.300521:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788201975.300559:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788201975.300574:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788201975.300627:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788201975.300640:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788201975.300650:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788201975.300661:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788201975.300669:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201972/real 1788201975] req@ffff8e828793c000 x1875062187648128/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201975.300695:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187648128/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201975.300705:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201975.300709:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788201975.300712:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187648128:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788201975.300718:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201972/real 1788201975] req@ffff8e828793f480 x1875062187648256/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201975.300726:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187648256/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788201997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201975.300731:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201975.300733:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788201975.300735:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187648256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788201975.300739:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201972/real 1788201975] req@ffff8e828793dc00 x1875062187648000/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201975.300752:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187648000/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788201997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201975.300760:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201975.300763:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788201975.300767:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187648000:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788201975.300791:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201972/real 1788201975] req@ffff8e828793c700 x1875062187648512/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201975.300803:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187648512/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201975.300811:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201975.300814:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788201975.300818:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187648512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788201975.300826:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201972/real 1788201975] req@ffff8e828793ca80 x1875062187648384/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201975.300839:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187648384/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788201997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201975.300846:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201975.300850:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788201975.300853:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187648384:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201976.004492:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201977.028490:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201977.348545:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201977.348557:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201977.348561:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201977.348583:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3432 00000100:00080000:1.0:1788201977.348610:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201977.348620:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788201977.348670:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187648640/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201977.348705:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201977.348710:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201977.348711:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201977.348716:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3432 00000100:00080000:1.0:1788201977.348720:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201977.348725:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201977.348737:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201977.348753:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187648768/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201977.348770:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201977.348774:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788201977.348776:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201977.348780:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3432 00000100:00080000:1.0:1788201977.348784:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788201977.348789:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187648640:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788201977.348792:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201977.348802:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201977.348816:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187648896/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201977.348827:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201977.348830:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201977.348832:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201977.348835:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3432 00000100:00080000:1.0:1788201977.348838:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201977.348842:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201977.348846:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788201977.348855:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187649024/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201977.348862:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201977.348866:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201977.348869:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201977.348873:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3432 00000100:00080000:1.0:1788201977.348878:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201977.348882:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201977.348888:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788201977.348894:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788201977.348904:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187649152/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201977.348957:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201977/real 1788201977] req@ffff8e828d722680 x1875062187648640/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201977.348980:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187648640/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201977.348995:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201977.349000:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201977.349005:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187648640:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788201977.349014:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187648768:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201977.349042:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201977.349057:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201977/real 1788201977] req@ffff8e828d723800 x1875062187648768/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201977.349070:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187648768/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201977.349078:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201977.349081:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201977.349084:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187648768:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201977.349118:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187648896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201977.349138:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201977.349151:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201977/real 1788201977] req@ffff8e828d721500 x1875062187648896/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201977.349170:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187648896/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201977.349179:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201977.349182:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201977.349186:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187648896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201977.349191:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187649024:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201977.349214:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201977.349225:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201977/real 1788201977] req@ffff8e828d720000 x1875062187649024/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201977.349236:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187649024/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201977.349243:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201977.349246:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201977.349249:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187649024:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201977.349255:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187649152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201977.349386:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201977.349405:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201977/real 1788201977] req@ffff8e828d720e00 x1875062187649152/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201977.349419:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187649152/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201977.349432:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201977.349436:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201977.349440:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187649152:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201978.052434:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201979.076458:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201980.100504:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201981.124437:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201982.148515:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788201982.468412:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201982.468422:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788201982.468426:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201982.468434:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3437 00000100:00080000:2.0:1788201982.468443:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201982.468449:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788201982.468502:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187649280/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201982.468525:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201982.468527:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788201982.468529:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201982.468532:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3437 00000100:00080000:2.0:1788201982.468535:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201982.468537:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201982.468545:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201982.468556:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187649408/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201982.468564:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201982.468567:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788201982.468571:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788201982.468572:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187649280:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788201982.468576:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3437 00000100:00080000:2.0:1788201982.468581:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201982.468589:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201982.468593:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201982.468602:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187649536/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201982.468608:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201982.468611:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788201982.468612:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788201982.468615:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3437 00000100:00080000:2.0:1788201982.468617:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201982.468619:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201982.468621:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201982.468628:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187649664/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788201982.468633:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788201982.468635:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788201982.468637:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788201982.468640:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788201982.468644:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3437 00000100:00080000:2.0:1788201982.468647:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788201982.468651:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788201982.468655:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201982.468666:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187649792/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788201982.468680:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201982/real 1788201982] req@ffff8e82894e5880 x1875062187649280/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201982.468695:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187649280/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201982.468703:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201982.468706:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788201982.468709:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187649280:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788201982.468726:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187649408:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788201982.468740:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788201982.468747:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201982/real 1788201982] req@ffff8e82894e4000 x1875062187649408/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201982.468765:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187649408/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201982.468771:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201982.468773:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788201982.468775:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187649408:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788201982.468778:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187649536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788201982.468790:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788201982.468797:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201982/real 1788201982] req@ffff8e82894e6a00 x1875062187649536/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201982.468804:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187649536/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201982.468808:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201982.468810:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788201982.468812:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187649536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788201982.468815:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187649664:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788201982.468825:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788201982.468830:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201982/real 1788201982] req@ffff8e82894e6680 x1875062187649664/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201982.468837:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187649664/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201982.468847:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201982.468849:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788201982.468851:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187649664:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788201982.468854:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187649792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788201982.468863:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788201982.468868:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201982/real 1788201982] req@ffff8e82894e7480 x1875062187649792/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788201982.468874:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187649792/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788201982.468879:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788201982.468881:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788201982.468883:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187649792:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201983.172449:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201984.196503:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201985.220538:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201986.244526:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201987.268499:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788201987.588487:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201987.588502:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788201987.588509:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201987.588523:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3442 00000100:00080000:1.0:1788201987.588546:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201987.588557:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788201987.588613:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187649920/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201987.588662:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201987.588665:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788201987.588667:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201987.588671:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3442 00000100:00080000:1.0:1788201987.588675:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201987.588679:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201987.588697:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201987.588718:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187650048/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201987.588725:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187649920:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788201987.588739:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201987.588743:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788201987.588744:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201987.588748:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3442 00000100:00080000:1.0:1788201987.588751:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201987.588755:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201987.588760:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788201987.588770:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187650176/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788201987.588778:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201987.588780:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788201987.588781:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201987.588785:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3442 00000100:00080000:1.0:1788201987.588790:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201987.588794:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201987.588800:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788201987.588813:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187650304/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788201987.588820:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788201987.588830:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788201987.588834:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788201987.588837:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788201987.588843:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3442 00000100:00080000:1.0:1788201987.588848:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788201987.588851:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788201987.588856:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788201987.588867:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187650432/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788201987.588877:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201987/real 1788201987] req@ffff8e828d722d80 x1875062187649920/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201987.588894:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187649920/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201987.588905:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201987.588909:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201987.588912:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187649920:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788201987.588936:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187650048:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201987.588953:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201987.588971:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201987/real 1788201987] req@ffff8e828d721180 x1875062187650048/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201987.588980:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187650048/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201987.588985:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201987.588988:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201987.588990:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187650048:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201987.588995:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187650176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201987.589007:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201987.589014:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201987/real 1788201987] req@ffff8e828d720a80 x1875062187650176/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201987.589022:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187650176/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201987.589027:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201987.589029:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201987.589031:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187650176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201987.589034:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187650304:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201987.589046:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201987.589052:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201987/real 1788201987] req@ffff8e828d720380 x1875062187650304/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201987.589059:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187650304/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201987.589064:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201987.589066:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201987.589067:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187650304:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201987.589071:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187650432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201987.589080:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201987.589085:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201987/real 1788201987] req@ffff8e828d722300 x1875062187650432/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201987.589093:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187650432/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201987.589098:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201987.589100:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201987.589101:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187650432:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201988.292550:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201989.316570:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201990.340528:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201991.364484:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201992.388494:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201992.708428:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201992.708438:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201992.708443:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201992.708453:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3447 00000100:00080000:0.0:1788201992.708464:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201992.708471:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201992.708511:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187650560/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201992.708539:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201992.708543:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201992.708545:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201992.708550:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3447 00000100:00080000:0.0:1788201992.708556:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201992.708560:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788201992.708569:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187650560:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788201992.708570:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201992.708590:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187650688/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201992.708611:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201992.708615:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201992.708617:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201992.708621:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3447 00000100:00080000:0.0:1788201992.708626:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788201992.708632:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788201992.708633:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201992.708640:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201992.708653:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187650816/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201992.708664:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201992.708668:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201992.708672:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788201992.708672:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201992/real 1788201992] req@ffff8e828793f100 x1875062187650560/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201992.708683:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3447 00000100:00080000:0.0:1788201992.708690:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788201992.708695:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187650560/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201992.708697:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201992.708707:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788201992.708709:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201992.708715:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201992.708721:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187650560:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788201992.708724:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187650944/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201992.708736:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201992.708739:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788201992.708740:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201992.708744:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3447 00000100:00080000:0.0:1788201992.708747:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201992.708751:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201992.708756:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788201992.708758:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187650688:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788201992.708769:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187651072/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788201992.708789:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201992.708802:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201992/real 1788201992] req@ffff8e828793ce00 x1875062187650688/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201992.708812:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187650688/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201992.708822:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201992.708824:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201992.708827:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187650688:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201992.708831:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187650816:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788201992.708842:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201992.708848:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201992/real 1788201992] req@ffff8e828793f800 x1875062187650816/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201992.708855:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187650816/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201992.708861:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201992.708862:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201992.708864:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187650816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788201992.708868:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187650944:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201992.708878:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201992.708884:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201992/real 1788201992] req@ffff8e828793e680 x1875062187650944/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201992.708891:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187650944/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201992.708896:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201992.708898:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201992.708901:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187650944:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788201992.708912:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187651072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788201992.708922:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788201992.708928:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201992/real 1788201992] req@ffff8e828793e300 x1875062187651072/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788201992.708935:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187651072/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788201992.708940:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788201992.708942:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788201992.708944:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187651072:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788201993.412502:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201994.436485:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788201995.460456:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201996.484500:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201997.508448:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788201997.828459:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201997.828471:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788201997.828478:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201997.828491:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3452 00000100:00080000:0.0:1788201997.828506:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201997.828516:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788201997.828565:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187651200/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201997.828609:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201997.828614:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788201997.828617:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201997.828623:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3452 00000100:00080000:0.0:1788201997.828630:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201997.828635:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201997.828648:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788201997.828649:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187651200:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788201997.828668:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187651328/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201997.828683:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201997.828688:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788201997.828690:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201997.828695:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3452 00000100:00080000:0.0:1788201997.828701:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201997.828707:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201997.828713:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788201997.828730:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187651456/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788201997.828732:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788201997.828751:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201997.828755:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788201997.828759:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201997.828764:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3452 00000100:00080000:0.0:1788201997.828770:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201997.828775:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788201997.828780:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788201997.828788:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201997/real 1788201997] req@ffff8e828793e300 x1875062187651200/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788201997.828797:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187651584/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788201997.828833:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788201997.828839:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788201997.828841:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187651200/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788201997.828843:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788201997.828859:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3452 00000100:00080000:0.0:1788201997.828872:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788201997.828886:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:2.0:1788201997.828889:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788201997.828897:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788201997.828911:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788201997.828916:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187651712/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788201997.828925:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187651200:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788201997.828971:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187651328:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201997.829000:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201997.829015:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201997/real 1788201997] req@ffff8e828793e680 x1875062187651328/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201997.829029:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187651328/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201997.829040:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201997.829044:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201997.829049:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187651328:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201997.829057:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187651456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788201997.829085:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201997.829104:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201997/real 1788201997] req@ffff8e828793f800 x1875062187651456/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201997.829121:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187651456/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201997.829131:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201997.829139:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201997.829145:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187651456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788201997.829160:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187651584:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201997.829180:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201997.829189:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201997/real 1788201997] req@ffff8e828793ce00 x1875062187651584/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201997.829196:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187651584/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201997.829202:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201997.829212:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201997.829283:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187651584:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788201997.829288:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187651712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788201997.829305:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788201997.829342:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788201997/real 1788201997] req@ffff8e828793f100 x1875062187651712/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788201997.829351:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187651712/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788201997.829357:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788201997.829361:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788201997.829364:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187651712:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788201998.532456:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788201999.556561:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202000.580498:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202001.604503:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202002.628557:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202002.948511:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202002.948524:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202002.948531:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202002.948544:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3458 00000100:00080000:2.0:1788202002.948558:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202002.948566:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202002.948620:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187651840/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202002.948654:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202002.948657:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202002.948659:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202002.948663:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3458 00000100:00080000:2.0:1788202002.948666:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202002.948668:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202002.948678:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202002.948688:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187651968/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202002.948701:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202002.948706:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202002.948716:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202002.948721:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3458 00000100:00080000:2.0:1788202002.948725:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202002.948730:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202002.948737:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202002.948748:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187651840:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202002.948761:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187652096/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202002.948779:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202002.948783:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202002.948785:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202002.948790:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3458 00000100:00080000:2.0:1788202002.948793:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202002.948797:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202002.948801:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202002.948818:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187652224/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202002.948828:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202002.948834:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202002.948837:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202002.948842:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3458 00000100:00080000:2.0:1788202002.948850:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788202002.948854:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788202002.948863:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202002.948871:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202002.948887:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187652352/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202002.948936:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202002/real 1788202002] req@ffff8e82894e4e00 x1875062187651840/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202002.948967:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187651840/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202002.948983:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202002.948989:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202002.948995:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187651840:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202002.949005:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187651968:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202002.949035:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202002.949052:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202002/real 1788202002] req@ffff8e82894e7800 x1875062187651968/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202002.949067:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187651968/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202002.949077:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202002.949081:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202002.949085:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187651968:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202002.949115:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187652096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202002.949186:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202002.949202:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202002/real 1788202002] req@ffff8e82894e6300 x1875062187652096/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202002.949218:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187652096/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202002.949229:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202002.949233:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202002.949238:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187652096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202002.949246:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187652224:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202002.949268:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202002.949280:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202002/real 1788202002] req@ffff8e82894e4a80 x1875062187652224/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202002.949294:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187652224/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202002.949304:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202002.949340:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202002.949345:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187652224:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202002.949353:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187652352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202002.949386:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202002.949410:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202002/real 1788202002] req@ffff8e82894e6d80 x1875062187652352/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202002.949435:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187652352/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202002.949452:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202002.949458:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202002.949465:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187652352:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202003.652586:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202004.676587:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202005.700520:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202006.724510:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202007.748517:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202008.068460:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202008.068477:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202008.068482:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202008.068496:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3463 00000100:00080000:1.0:1788202008.068511:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202008.068518:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202008.068566:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187652480/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202008.068594:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202008.068598:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202008.068599:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202008.068603:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3463 00000100:00080000:1.0:1788202008.068605:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202008.068608:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202008.068619:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202008.068633:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187652608/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202008.068647:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788202008.068647:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187652480:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202008.068659:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202008.068661:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202008.068666:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3463 00000100:00080000:1.0:1788202008.068670:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202008.068675:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202008.068681:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202008.068692:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187652736/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202008.068699:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202008.068702:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202008.068704:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202008.068707:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3463 00000100:00080000:1.0:1788202008.068709:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202008.068712:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202008.068716:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202008.068723:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187652864/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202008.068729:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202008.068735:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1788202008.068735:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788202008.068740:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202008.068743:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3463 00000100:00080000:1.0:1788202008.068745:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202008.068748:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202008.068752:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202008.068760:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187652992/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202008.068788:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202008/real 1788202008] req@ffff8e828d722300 x1875062187652480/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202008.068804:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187652480/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202008.068813:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202008.068817:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202008.068820:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187652480:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202008.068843:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187652608:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202008.068859:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202008.068868:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202008/real 1788202008] req@ffff8e828d720380 x1875062187652608/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202008.068876:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187652608/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202008.068882:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202008.068884:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202008.068887:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187652608:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202008.068890:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187652736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202008.068900:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202008.068915:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202008/real 1788202008] req@ffff8e828d720a80 x1875062187652736/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202008.068923:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187652736/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202008.068929:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202008.068931:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202008.068933:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187652736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202008.068937:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187652864:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202008.068950:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202008.068956:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202008/real 1788202008] req@ffff8e828d721180 x1875062187652864/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202008.068965:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187652864/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202008.068970:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202008.068972:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202008.068973:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187652864:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202008.068977:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187652992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202008.068987:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202008.068992:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202008/real 1788202008] req@ffff8e828d722d80 x1875062187652992/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202008.069000:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187652992/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202008.069005:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202008.069007:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202008.069008:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187652992:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202008.772460:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202009.796494:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202010.820522:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202011.844508:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202012.868479:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202013.188419:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202013.188433:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202013.188436:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202013.188445:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3468 00000100:00080000:1.0:1788202013.188455:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202013.188461:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202013.188494:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187653120/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202013.188523:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202013.188525:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202013.188527:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202013.188530:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3468 00000100:00080000:1.0:1788202013.188532:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202013.188535:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202013.188542:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202013.188549:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187653248/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202013.188559:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202013.188561:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202013.188563:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202013.188566:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3468 00000100:00080000:1.0:1788202013.188568:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202013.188571:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202013.188575:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202013.188582:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187653376/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202013.188589:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788202013.188590:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187653120:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202013.188592:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202013.188595:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202013.188601:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3468 00000100:00080000:1.0:1788202013.188604:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202013.188606:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202013.188609:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202013.188617:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187653504/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202013.188623:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202013.188625:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202013.188626:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202013.188629:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3468 00000100:00080000:1.0:1788202013.188632:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202013.188633:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202013.188637:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202013.188643:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187653632/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202013.188667:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202013.188716:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202013/real 1788202013] req@ffff8e828d720e00 x1875062187653120/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202013.188732:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187653120/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202013.188745:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202013.188748:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202013.188751:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187653120:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202013.188756:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187653248:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202013.188773:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202013.188782:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202013/real 1788202013] req@ffff8e828d720000 x1875062187653248/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202013.188790:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187653248/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202013.188795:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202013.188797:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202013.188799:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187653248:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202013.188819:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187653376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202013.188830:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202013.188837:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202013/real 1788202013] req@ffff8e828d721500 x1875062187653376/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202013.188844:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187653376/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202013.188849:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202013.188851:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202013.188853:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187653376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202013.188857:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187653504:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202013.188867:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202013.188872:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202013/real 1788202013] req@ffff8e828d723800 x1875062187653504/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202013.188879:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187653504/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202013.188884:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202013.188886:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202013.188888:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187653504:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202013.188891:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187653632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202013.188900:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202013.188906:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202013/real 1788202013] req@ffff8e828d722680 x1875062187653632/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202013.188912:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187653632/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202013.188917:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202013.188919:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202013.188921:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187653632:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202013.892491:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202014.916474:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202015.940486:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202016.964461:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202017.988565:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202018.308441:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202018.308456:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202018.308462:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202018.308476:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3473 00000100:00080000:1.0:1788202018.308491:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202018.308501:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202018.308577:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187653760/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202018.308604:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202018.308607:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202018.308609:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202018.308612:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3473 00000100:00080000:1.0:1788202018.308616:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202018.308619:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202018.308629:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202018.308638:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187653888/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202018.308662:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202018.308665:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202018.308666:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202018.308669:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3473 00000100:00080000:1.0:1788202018.308671:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202018.308674:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202018.308678:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202018.308685:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187654016/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202018.308692:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202018.308694:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202018.308696:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202018.308700:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3473 00000100:00080000:1.0:1788202018.308703:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202018.308705:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202018.308709:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202018.308715:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187654144/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202018.308725:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202018.308727:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202018.308728:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202018.308731:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3473 00000100:00080000:1.0:1788202018.308733:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202018.308739:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202018.308743:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187653760:192.168.201.121@tcp:250:kworker.0 00000008:00100000:1.0:1788202018.308745:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202018.308762:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187654272/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202018.308820:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202018.308894:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202018/real 1788202018] req@ffff8e828d722680 x1875062187653760/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202018.308910:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187653760/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202018.308919:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202018.308923:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202018.308926:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187653760:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202018.308931:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187653888:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202018.308947:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202018.308956:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202018/real 1788202018] req@ffff8e828d723800 x1875062187653888/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202018.308963:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187653888/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202018.308968:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202018.308971:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202018.308973:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187653888:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202018.308976:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187654016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202018.308986:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202018.308992:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202018/real 1788202018] req@ffff8e828d721500 x1875062187654016/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202018.308999:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187654016/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202018.309009:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202018.309011:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202018.309013:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187654016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202018.309034:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187654144:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202018.309045:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202018.309051:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202018/real 1788202018] req@ffff8e828d720000 x1875062187654144/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202018.309058:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187654144/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202018.309063:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202018.309065:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202018.309066:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187654144:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202018.309070:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187654272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202018.309080:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202018.309086:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202018/real 1788202018] req@ffff8e828d720e00 x1875062187654272/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202018.309093:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187654272/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202018.309097:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202018.309099:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202018.309102:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187654272:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202019.012560:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202020.036539:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202021.060594:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202022.084483:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202023.108586:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202023.428408:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202023.428419:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202023.428425:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202023.428436:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3478 00000100:00080000:2.0:1788202023.428449:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202023.428457:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202023.428514:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187654400/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202023.428539:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202023.428543:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202023.428545:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202023.428548:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3478 00000100:00080000:2.0:1788202023.428552:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202023.428555:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202023.428564:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202023.428576:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187654528/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202023.428587:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202023.428589:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202023.428591:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202023.428594:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3478 00000100:00080000:2.0:1788202023.428596:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202023.428604:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202023.428607:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187654400:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788202023.428608:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202023.428622:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187654656/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202023.428629:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202023.428632:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202023.428633:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202023.428636:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3478 00000100:00080000:2.0:1788202023.428642:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202023.428645:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202023.428649:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202023.428659:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187654784/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202023.428665:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202023.428667:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202023.428668:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202023.428670:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3478 00000100:00080000:2.0:1788202023.428676:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202023.428680:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202023.428684:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788202023.428695:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202023.428697:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187654912/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202023.428758:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202023/real 1788202023] req@ffff8e82894e7100 x1875062187654400/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202023.428796:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187654400/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202023.428808:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202023.428813:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202023.428817:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187654400:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202023.428841:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187654528:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202023.428864:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202023.428876:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202023/real 1788202023] req@ffff8e82894e4380 x1875062187654528/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202023.428889:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187654528/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202023.428898:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202023.428901:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202023.428905:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187654528:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202023.428911:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187654656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202023.428927:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202023.428937:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202023/real 1788202023] req@ffff8e82894e5c00 x1875062187654656/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202023.428950:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187654656/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202023.428959:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202023.428961:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202023.428965:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187654656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202023.428971:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187654784:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202023.428990:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202023.429000:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202023/real 1788202023] req@ffff8e82894e5f80 x1875062187654784/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202023.429013:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187654784/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202023.429022:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202023.429025:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202023.429029:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187654784:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202023.429035:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187654912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202023.429050:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202023.429059:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202023/real 1788202023] req@ffff8e82894e6d80 x1875062187654912/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202023.429071:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187654912/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202023.429079:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202023.429083:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202023.429086:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187654912:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202024.132487:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202025.156512:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202026.180563:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202027.204536:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202028.228578:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202028.548409:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202028.548418:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202028.548422:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202028.548432:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3483 00000100:00080000:2.0:1788202028.548443:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202028.548451:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202028.548487:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187655040/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202028.548512:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202028.548515:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202028.548517:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202028.548520:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3483 00000100:00080000:2.0:1788202028.548522:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202028.548525:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202028.548533:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202028.548540:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187655168/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202028.548547:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202028.548549:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202028.548550:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202028.548552:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3483 00000100:00080000:2.0:1788202028.548554:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202028.548557:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202028.548560:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202028.548566:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187655296/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202028.548572:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202028.548574:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202028.548575:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202028.548578:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3483 00000100:00080000:2.0:1788202028.548580:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788202028.548585:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187655040:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202028.548587:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202028.548596:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202028.548609:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187655424/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202028.548620:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202028.548625:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202028.548627:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202028.548631:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3483 00000100:00080000:2.0:1788202028.548635:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202028.548640:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202028.548645:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788202028.548650:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202028.548658:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187655552/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202028.548725:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202028/real 1788202028] req@ffff8e82894e4a80 x1875062187655040/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202028.548745:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187655040/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202028.548755:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202028.548759:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202028.548763:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187655040:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202028.548770:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187655168:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202028.548793:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202028.548805:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202028/real 1788202028] req@ffff8e82894e6300 x1875062187655168/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202028.548816:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187655168/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202028.548824:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202028.548827:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202028.548831:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187655168:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202028.548837:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187655296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202028.548853:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202028.548864:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202028/real 1788202028] req@ffff8e82894e7800 x1875062187655296/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202028.548875:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187655296/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202028.548883:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202028.548886:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202028.548889:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187655296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202028.548913:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187655424:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202028.548934:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202028.548946:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202028/real 1788202028] req@ffff8e82894e4e00 x1875062187655424/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202028.548956:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187655424/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202028.548962:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202028.548964:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202028.548966:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187655424:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202028.548969:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187655552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202028.548983:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202028.548989:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202028/real 1788202028] req@ffff8e82894e7480 x1875062187655552/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202028.548997:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187655552/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202028.549002:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202028.549004:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202028.549006:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187655552:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202029.252535:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202030.276577:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202031.300481:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202032.324439:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202033.348490:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202033.668434:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202033.668446:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202033.668451:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202033.668462:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3488 00000100:00080000:2.0:1788202033.668474:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202033.668483:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202033.668522:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187655680/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202033.668551:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202033.668557:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202033.668559:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202033.668563:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3488 00000100:00080000:2.0:1788202033.668567:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202033.668571:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202033.668580:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202033.668605:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187655808/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202033.668616:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187655680:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202033.668622:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202033.668625:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202033.668627:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202033.668631:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3488 00000100:00080000:2.0:1788202033.668635:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202033.668640:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202033.668644:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202033.668656:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187655936/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202033.668663:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202033.668666:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202033.668668:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202033.668671:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3488 00000100:00080000:2.0:1788202033.668674:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788202033.668678:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788202033.668680:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202033.668687:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202033.668697:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187656064/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202033.668704:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202033.668706:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202033.668708:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202033.668711:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3488 00000100:00080000:2.0:1788202033.668714:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202033.668719:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202033/real 1788202033] req@ffff8e82894e6680 x1875062187655680/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202033.668719:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202033.668722:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202033.668735:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187656192/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202033.668742:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187655680/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202033.668752:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202033.668755:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202033.668760:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187655680:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202033.668777:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187655808:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202033.668792:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202033.668803:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202033/real 1788202033] req@ffff8e82894e6a00 x1875062187655808/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202033.668812:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187655808/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202033.668823:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202033.668825:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202033.668828:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187655808:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202033.668833:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187655936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202033.668845:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202033.668851:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202033/real 1788202033] req@ffff8e82894e4000 x1875062187655936/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202033.668859:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187655936/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202033.668864:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202033.668866:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202033.668867:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187655936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202033.668871:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187656064:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202033.668883:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202033.668888:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202033/real 1788202033] req@ffff8e82894e5880 x1875062187656064/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202033.668896:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187656064/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202033.668901:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202033.668903:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202033.668905:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187656064:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202033.668909:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187656192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202033.668919:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202033.668926:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202033/real 1788202033] req@ffff8e82894e7480 x1875062187656192/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202033.668934:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187656192/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202033.668939:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202033.668940:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202033.668943:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187656192:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202034.372452:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202035.396519:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202036.420530:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202037.444489:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202038.468493:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202038.788405:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202038.788413:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202038.788417:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202038.788426:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3493 00000100:00080000:3.0:1788202038.788434:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202038.788440:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202038.788472:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187656320/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202038.788499:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202038.788515:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202038.788517:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202038.788520:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3493 00000100:00080000:3.0:1788202038.788527:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202038.788531:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202038.788539:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202038.788550:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187656448/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202038.788555:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187656320:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202038.788573:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202038.788577:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202038.788579:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202038.788584:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3493 00000100:00080000:3.0:1788202038.788588:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202038.788592:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202038.788598:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202038.788611:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187656576/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202038.788620:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202038.788632:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202038.788635:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202038.788643:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3493 00000100:00080000:3.0:1788202038.788648:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202038.788653:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202038.788659:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202038.788669:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187656448:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202038.788677:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187656704/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202038.788689:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202038.788693:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202038.788695:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202038.788699:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3493 00000100:00080000:3.0:1788202038.788704:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202038.788712:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187656576:192.168.201.121@tcp:38:kworker.0 00000100:00080000:3.0:1788202038.788712:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202038.788722:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202038.788737:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187656832/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202038.788748:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187656704:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202038.788771:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187656832:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202039.492560:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202040.516501:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202041.540524:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788202041.860693:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:2.0:1788202041.860741:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788202041.860761:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788202041.860820:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788202041.860833:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788202041.860845:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788202041.860858:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788202041.860874:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202038/real 1788202041] req@ffff8e8289539f80 x1875062187656704/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202041.860907:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187656704/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202041.860921:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202041.860928:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202041.860933:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187656704:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202041.860941:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202038/real 1788202041] req@ffff8e8289538380 x1875062187656448/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202041.860955:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187656448/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202041.860964:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202041.860968:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202041.860971:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187656448:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202041.860978:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202038/real 1788202041] req@ffff8e828953b100 x1875062187656576/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202041.860990:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187656576/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202041.860997:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202041.861001:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202041.861004:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187656576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202041.861010:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202038/real 1788202041] req@ffff8e828953bb80 x1875062187656320/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202041.861022:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187656320/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202041.861030:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202041.861034:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202041.861038:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187656320:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202041.861073:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202038/real 1788202041] req@ffff8e8289539500 x1875062187656832/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202041.861086:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187656832/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202041.861094:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202041.861098:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202041.861101:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187656832:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202042.564571:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202043.588543:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202043.908464:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202043.908474:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202043.908478:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202043.908489:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3498 00000100:00080000:0.0:1788202043.908501:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202043.908510:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202043.908550:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187656960/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202043.908568:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202043.908571:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202043.908573:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202043.908577:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3498 00000100:00080000:0.0:1788202043.908580:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202043.908586:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202043.908588:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187656960:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788202043.908594:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202043.908606:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187657088/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202043.908617:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202043.908621:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202043.908623:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202043.908627:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3498 00000100:00080000:0.0:1788202043.908653:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788202043.908657:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202043.908662:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202043.908670:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202043.908685:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187657216/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202043.908696:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202043.908699:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202043.908702:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788202043.908707:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202043/real 1788202043] req@ffff8e828793ca80 x1875062187656960/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202043.908710:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3498 00000100:00080000:0.0:1788202043.908718:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202043.908725:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202043.908732:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202043.908737:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187656960/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202043.908750:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187657344/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202043.908751:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202043.908762:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:0.0:1788202043.908768:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202043.908776:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1788202043.908779:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187656960:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202043.908780:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202043.908790:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3498 00000100:00080000:0.0:1788202043.908795:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202043.908799:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202043.908804:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202043.908814:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187657088:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202043.908820:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187657472/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202043.908855:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202043.908872:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202043/real 1788202043] req@ffff8e828793c700 x1875062187657088/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202043.908886:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187657088/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202043.908894:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202043.908897:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202043.908901:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187657088:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202043.908907:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187657216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202043.908929:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202043.908942:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202043/real 1788202043] req@ffff8e828793dc00 x1875062187657216/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202043.908972:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187657216/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202043.908981:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202043.908984:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202043.908988:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187657216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202043.908994:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187657344:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202043.909017:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202043.909030:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202043/real 1788202043] req@ffff8e828793f480 x1875062187657344/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202043.909041:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187657344/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202043.909050:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202043.909053:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202043.909056:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187657344:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202043.909072:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187657472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202043.909093:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202043.909105:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202043/real 1788202043] req@ffff8e828793c000 x1875062187657472/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202043.909116:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187657472/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202043.909124:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202043.909127:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202043.909130:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187657472:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202044.612548:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202045.636476:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202046.660444:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202047.684478:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202048.708505:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202049.028418:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202049.028426:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202049.028429:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202049.028438:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3504 00000100:00080000:2.0:1788202049.028454:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202049.028461:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202049.028492:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187657600/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202049.028515:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202049.028518:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202049.028519:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202049.028522:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3504 00000100:00080000:2.0:1788202049.028524:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202049.028527:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202049.028535:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202049.028548:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187657728/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202049.028559:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202049.028563:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202049.028566:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202049.028570:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3504 00000100:00080000:2.0:1788202049.028575:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202049.028579:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202049.028585:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202049.028597:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187657856/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202049.028605:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187657600:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202049.028609:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202049.028621:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202049.028624:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202049.028627:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3504 00000100:00080000:2.0:1788202049.028630:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202049.028633:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202049.028636:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202049.028654:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187657984/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202049.028667:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202049.028671:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202049.028673:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202049.028677:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3504 00000100:00080000:2.0:1788202049.028681:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202049.028686:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202049.028691:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202049.028703:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187658112/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202049.028720:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202049.028782:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202049/real 1788202049] req@ffff8e82894e4e00 x1875062187657600/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202049.028830:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187657600/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202049.028846:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202049.028849:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202049.028853:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187657600:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202049.028860:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187657728:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202049.028879:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202049.028889:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202049/real 1788202049] req@ffff8e82894e7800 x1875062187657728/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202049.028899:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187657728/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202049.028907:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202049.028911:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202049.028915:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187657728:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202049.028946:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187657856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202049.028965:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202049.028973:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202049/real 1788202049] req@ffff8e82894e6300 x1875062187657856/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202049.028982:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187657856/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202049.028987:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202049.028990:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202049.028992:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187657856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202049.028997:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187657984:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202049.029010:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202049.029025:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202049/real 1788202049] req@ffff8e82894e4a80 x1875062187657984/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202049.029035:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187657984/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202049.029041:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202049.029043:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202049.029046:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187657984:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202049.029049:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187658112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202049.029061:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202049.029067:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202049/real 1788202049] req@ffff8e82894e6d80 x1875062187658112/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202049.029074:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187658112/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202049.029080:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202049.029082:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202049.029083:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187658112:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202049.732485:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202050.756480:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202051.780482:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202052.804502:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202053.828457:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202054.148414:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202054.148430:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202054.148435:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202054.148445:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3509 00000100:00080000:2.0:1788202054.148457:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202054.148464:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202054.148501:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187658240/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202054.148540:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202054.148545:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202054.148547:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202054.148552:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3509 00000100:00080000:2.0:1788202054.148562:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202054.148566:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202054.148575:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202054.148589:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187658368/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202054.148598:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187658240:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202054.148604:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202054.148609:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202054.148611:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202054.148616:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3509 00000100:00080000:2.0:1788202054.148621:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202054.148625:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202054.148630:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202054.148642:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187658496/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202054.148652:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202054.148656:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202054.148658:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202054.148662:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3509 00000100:00080000:2.0:1788202054.148667:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202054.148672:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202054.148677:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788202054.148683:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202054.148692:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187658624/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202054.148703:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202054.148707:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202054.148708:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202054.148712:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3509 00000100:00080000:2.0:1788202054.148716:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202054.148720:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202054.148725:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202054.148738:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187658752/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202054.148743:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202054/real 1788202054] req@ffff8e82894e5f80 x1875062187658240/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202054.148771:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187658240/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202054.148783:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202054.148788:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202054.148792:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187658240:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202054.148816:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187658368:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202054.148837:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202054.148847:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202054/real 1788202054] req@ffff8e82894e5c00 x1875062187658368/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202054.148856:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187658368/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202054.148861:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202054.148865:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202054.148869:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187658368:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202054.148874:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187658496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202054.148887:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202054.148895:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202054/real 1788202054] req@ffff8e82894e4380 x1875062187658496/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202054.148902:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187658496/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202054.148908:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202054.148910:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202054.148912:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187658496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202054.148916:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187658624:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202054.148926:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202054.148932:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202054/real 1788202054] req@ffff8e82894e7100 x1875062187658624/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202054.148939:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187658624/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202054.148944:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202054.148946:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202054.148947:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187658624:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202054.148951:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187658752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202054.148960:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202054.148966:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202054/real 1788202054] req@ffff8e82894e6d80 x1875062187658752/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202054.148973:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187658752/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202054.148978:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202054.148980:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202054.148982:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187658752:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202054.852499:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202055.876484:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202056.900473:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202057.924497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202058.948471:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202059.268426:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202059.268434:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202059.268438:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202059.268446:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3514 00000100:00080000:2.0:1788202059.268455:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202059.268461:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202059.268498:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187658880/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202059.268521:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202059.268526:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202059.268527:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202059.268530:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3514 00000100:00080000:2.0:1788202059.268533:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202059.268535:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202059.268542:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202059.268550:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187659008/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202059.268556:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202059.268559:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202059.268560:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202059.268562:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3514 00000100:00080000:2.0:1788202059.268564:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202059.268567:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202059.268570:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202059.268575:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187659136/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202059.268583:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202059.268585:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202059.268587:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788202059.268591:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187658880:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202059.268592:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3514 00000100:00080000:2.0:1788202059.268595:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202059.268600:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202059.268605:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202059.268612:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187659264/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202059.268618:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202059.268620:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202059.268621:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202059.268624:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3514 00000100:00080000:2.0:1788202059.268626:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202059.268628:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202059.268631:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202059.268639:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187659392/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202059.268662:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202059.268705:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202059/real 1788202059] req@ffff8e82894e4a80 x1875062187658880/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202059.268719:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187658880/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202059.268729:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202059.268733:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202059.268736:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187658880:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202059.268742:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187659008:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202059.268758:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202059.268766:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202059/real 1788202059] req@ffff8e82894e6300 x1875062187659008/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202059.268775:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187659008/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202059.268785:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202059.268787:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202059.268789:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187659008:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202059.268807:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187659136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202059.268818:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202059.268824:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202059/real 1788202059] req@ffff8e82894e7800 x1875062187659136/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202059.268832:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187659136/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202059.268837:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202059.268839:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202059.268841:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187659136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202059.268844:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187659264:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202059.268854:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202059.268860:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202059/real 1788202059] req@ffff8e82894e4e00 x1875062187659264/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202059.268867:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187659264/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202059.268872:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202059.268874:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202059.268876:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187659264:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202059.268880:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187659392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202059.268890:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202059.268895:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202059/real 1788202059] req@ffff8e82894e7480 x1875062187659392/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202059.268902:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187659392/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202059.268909:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202059.268912:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202059.268916:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187659392:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202059.972654:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202060.996498:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202062.020498:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202063.044488:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202064.068578:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202064.388434:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202064.388443:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202064.388447:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202064.388455:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3519 00000100:00080000:0.0:1788202064.388464:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202064.388469:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202064.388507:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793df80 x1875062187659520/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202064.388529:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202064.388531:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202064.388533:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202064.388537:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3519 00000100:00080000:0.0:1788202064.388540:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202064.388543:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202064.388550:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202064.388558:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187659648/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202064.388567:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202064.388569:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202064.388570:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202064.388573:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3519 00000100:00080000:0.0:1788202064.388575:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202064.388577:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202064.388581:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202064.388586:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187659520:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202064.388591:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187659776/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202064.388599:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202064.388601:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202064.388602:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202064.388605:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3519 00000100:00080000:0.0:1788202064.388609:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202064.388612:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202064.388615:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202064.388625:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187659904/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202064.388633:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202064.388636:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202064.388637:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202064.388639:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3519 00000100:00080000:0.0:1788202064.388641:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202064.388644:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202064.388648:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788202064.388655:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202064.388656:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187660032/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202064.388693:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202064/real 1788202064] req@ffff8e828793df80 x1875062187659520/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202064.388708:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793df80 x1875062187659520/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202064.388716:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202064.388719:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202064.388722:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187659520:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202064.388727:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187659648:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202064.388742:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202064.388751:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202064/real 1788202064] req@ffff8e828793d500 x1875062187659648/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202064.388759:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187659648/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202064.388764:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202064.388767:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202064.388769:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187659648:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202064.388787:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187659776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202064.388806:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202064.388814:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202064/real 1788202064] req@ffff8e828793d880 x1875062187659776/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202064.388822:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187659776/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202064.388827:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202064.388829:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202064.388831:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187659776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202064.388835:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187659904:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202064.388845:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202064.388851:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202064/real 1788202064] req@ffff8e828793c380 x1875062187659904/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202064.388858:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187659904/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202064.388863:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202064.388865:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202064.388868:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187659904:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202064.388871:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187660032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202064.388880:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202064.388894:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202064/real 1788202064] req@ffff8e828793fb80 x1875062187660032/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202064.388906:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187660032/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202064.388914:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202064.388918:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202064.388921:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187660032:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202065.092450:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202066.116505:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202067.140473:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202068.164483:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202069.188465:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202069.508485:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202069.508494:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202069.508499:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202069.508510:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3524 00000100:00080000:0.0:1788202069.508520:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202069.508528:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202069.508564:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187660160/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202069.508602:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202069.508606:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202069.508609:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202069.508615:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3524 00000100:00080000:0.0:1788202069.508620:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202069.508625:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202069.508636:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202069.508647:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187660160:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202069.508656:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187660288/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202069.508670:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202069.508674:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202069.508677:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202069.508682:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3524 00000100:00080000:0.0:1788202069.508696:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202069.508704:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:1.0:1788202069.508704:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788202069.508711:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202069.508729:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187660416/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202069.508742:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202069.508747:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788202069.508748:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202069/real 1788202069] req@ffff8e828793fb80 x1875062187660160/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202069.508750:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202069.508761:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3524 00000100:00080000:0.0:1788202069.508767:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202069.508771:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187660160/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202069.508776:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:1.0:1788202069.508787:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788202069.508788:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788202069.508792:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202069.508797:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187660160:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202069.508807:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187660544/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202069.508818:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187660288:192.168.201.121@tcp:38:kworker.0 00000100:00080000:0.0:1788202069.508823:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202069.508828:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202069.508831:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202069.508837:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3524 00000800:00000100:1.0:1788202069.508840:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202069.508845:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202069.508852:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202069.508855:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202069/real 1788202069] req@ffff8e828793c380 x1875062187660288/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788202069.508860:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202069.508870:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187660288/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202069.508877:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788202069.508881:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793df80 x1875062187660672/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202069.508882:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202069.508887:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187660288:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202069.508896:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187660416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202069.508917:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202069.508927:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202069/real 1788202069] req@ffff8e828793d880 x1875062187660416/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202069.508941:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187660416/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202069.508950:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202069.508954:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202069.508958:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187660416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202069.508993:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187660544:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202069.509018:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202069.509030:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202069/real 1788202069] req@ffff8e828793d500 x1875062187660544/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202069.509052:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187660544/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202069.509061:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202069.509065:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202069.509068:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187660544:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202069.509075:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187660672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202069.509090:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202069.509100:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202069/real 1788202069] req@ffff8e828793df80 x1875062187660672/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202069.509108:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793df80 x1875062187660672/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202069.509114:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202069.509116:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202069.509118:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187660672:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202070.212468:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202071.236448:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202072.260536:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202073.284441:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202074.308486:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202074.628410:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202074.628419:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202074.628423:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202074.628433:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3529 00000100:00080000:3.0:1788202074.628442:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202074.628449:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202074.628481:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187660800/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202074.628507:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202074.628511:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202074.628513:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202074.628516:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3529 00000100:00080000:3.0:1788202074.628518:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202074.628521:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202074.628528:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202074.628537:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187660928/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202074.628544:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202074.628547:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202074.628548:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202074.628550:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3529 00000100:00080000:3.0:1788202074.628552:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202074.628555:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202074.628558:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202074.628569:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187661056/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202074.628570:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187660800:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202074.628585:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202074.628587:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202074.628588:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202074.628590:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3529 00000100:00080000:3.0:1788202074.628592:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202074.628594:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202074.628600:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202074.628614:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187661184/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202074.628624:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202074.628628:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202074.628630:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202074.628634:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3529 00000100:00080000:3.0:1788202074.628641:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202074.628645:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202074.628650:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788202074.628658:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202074.628664:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187661312/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202074.628705:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202074/real 1788202074] req@ffff8e8289538a80 x1875062187660800/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202074.628724:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187660800/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202074.628736:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202074.628741:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202074.628745:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187660800:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202074.628752:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187660928:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202074.628776:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202074.628790:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202074/real 1788202074] req@ffff8e828953a680 x1875062187660928/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202074.628803:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187660928/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202074.628811:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202074.628815:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202074.628818:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187660928:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202074.628842:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187661056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202074.628863:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202074.628875:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202074/real 1788202074] req@ffff8e8289539880 x1875062187661056/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202074.628908:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187661056/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202074.628917:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202074.628920:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202074.628924:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187661056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202074.628930:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187661184:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202074.628951:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202074.628963:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202074/real 1788202074] req@ffff8e8289539500 x1875062187661184/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202074.628982:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187661184/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202074.628990:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202074.628994:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202074.628997:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187661184:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202074.629002:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187661312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202074.629022:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202074.629033:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202074/real 1788202074] req@ffff8e828953bb80 x1875062187661312/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202074.629045:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187661312/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202074.629054:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202074.629057:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202074.629060:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187661312:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202075.332450:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202076.356482:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202077.380472:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202078.404443:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202079.428502:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202079.748417:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202079.748427:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202079.748432:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202079.748441:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3534 00000100:00080000:2.0:1788202079.748452:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202079.748459:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202079.748495:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187661440/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202079.748530:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202079.748534:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202079.748535:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202079.748540:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3534 00000100:00080000:2.0:1788202079.748543:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202079.748547:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202079.748556:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202079.748575:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187661568/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202079.748582:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187661440:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202079.748588:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202079.748595:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202079.748597:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202079.748601:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3534 00000100:00080000:2.0:1788202079.748604:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202079.748609:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202079.748614:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202079.748624:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187661696/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202079.748630:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202079.748632:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202079.748633:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202079.748636:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3534 00000100:00080000:2.0:1788202079.748639:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202079.748642:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202079.748645:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788202079.748656:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202079.748656:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187661824/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202079.748681:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202079.748684:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202079.748686:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202079.748690:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3534 00000100:00080000:2.0:1788202079.748695:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202079.748701:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202079/real 1788202079] req@ffff8e82894e5880 x1875062187661440/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202079.748702:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202079.748709:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202079.748727:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187661952/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202079.748736:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187661440/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202079.748746:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202079.748749:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202079.748752:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187661440:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202079.748776:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187661568:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202079.748824:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202079.748837:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202079/real 1788202079] req@ffff8e82894e4000 x1875062187661568/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202079.748845:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187661568/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202079.748850:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202079.748852:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202079.748854:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187661568:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202079.748858:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187661696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202079.748874:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202079.748894:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202079/real 1788202079] req@ffff8e82894e6a00 x1875062187661696/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202079.748912:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187661696/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202079.748918:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202079.748920:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202079.748922:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187661696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202079.748932:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187661824:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202079.748952:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202079.748963:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202079/real 1788202079] req@ffff8e82894e6680 x1875062187661824/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202079.748975:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187661824/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202079.748981:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202079.748983:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202079.748986:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187661824:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202079.748991:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187661952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202079.749010:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202079.749020:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202079/real 1788202079] req@ffff8e82894e7480 x1875062187661952/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202079.749032:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187661952/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202079.749043:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202079.749045:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202079.749047:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187661952:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202080.452451:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202081.476496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202082.500444:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202083.524472:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202084.548503:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202084.868432:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202084.868444:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202084.868448:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202084.868458:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3539 00000100:00080000:3.0:1788202084.868469:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202084.868474:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202084.868503:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187662080/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202084.868523:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202084.868526:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202084.868527:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202084.868530:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3539 00000100:00080000:3.0:1788202084.868533:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202084.868536:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202084.868543:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202084.868554:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187662208/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202084.868555:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187662080:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202084.868568:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202084.868570:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202084.868571:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202084.868573:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3539 00000100:00080000:3.0:1788202084.868575:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202084.868576:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202084.868579:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202084.868586:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187662336/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202084.868591:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202084.868593:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202084.868594:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202084.868596:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3539 00000100:00080000:3.0:1788202084.868598:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202084.868601:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202084.868604:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202084.868609:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187662464/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202084.868618:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202084.868620:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202084.868622:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788202084.868626:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788202084.868627:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3539 00000100:00080000:3.0:1788202084.868632:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202084.868634:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202084.868637:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202084.868643:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187662592/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202084.868674:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202084/real 1788202084] req@ffff8e828953b100 x1875062187662080/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202084.868689:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187662080/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202084.868696:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202084.868699:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202084.868722:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187662080:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202084.868744:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187662208:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202084.868766:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202084.868779:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202084/real 1788202084] req@ffff8e8289538380 x1875062187662208/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202084.868791:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187662208/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202084.868804:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202084.868807:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202084.868811:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187662208:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202084.868818:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187662336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202084.868837:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202084.868856:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202084/real 1788202084] req@ffff8e8289539f80 x1875062187662336/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202084.868867:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187662336/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202084.868875:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202084.868878:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202084.868881:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187662336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202084.868892:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187662464:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202084.868907:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202084.868914:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202084/real 1788202084] req@ffff8e828953bb80 x1875062187662464/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202084.868936:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187662464/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202084.868942:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202084.868944:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202084.868947:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187662464:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202084.868951:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187662592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202084.868963:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202084.868969:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202084/real 1788202084] req@ffff8e8289539500 x1875062187662592/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202084.868975:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187662592/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202084.868980:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202084.868982:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202084.868983:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187662592:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202085.572492:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202086.596478:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202087.620471:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202088.644444:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202089.668482:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202089.988382:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202089.988389:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202089.988393:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202089.988400:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3545 00000100:00080000:3.0:1788202089.988408:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202089.988413:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202089.988441:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187662720/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202089.988465:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202089.988468:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202089.988469:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202089.988471:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3545 00000100:00080000:3.0:1788202089.988473:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202089.988475:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202089.988483:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202089.988491:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187662848/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202089.988498:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202089.988500:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202089.988501:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202089.988509:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3545 00000100:00100000:1.0:1788202089.988510:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187662720:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202089.988519:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202089.988526:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202089.988532:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202089.988545:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187662976/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202089.988553:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202089.988555:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202089.988557:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202089.988559:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3545 00000800:00000100:1.0:1788202089.988563:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788202089.988564:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202089.988569:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202089.988573:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202089.988581:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187663104/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202089.988587:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202089.988589:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202089.988591:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202089.988594:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3545 00000100:00080000:3.0:1788202089.988596:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202089.988599:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202089/real 1788202089] req@ffff8e8289539880 x1875062187662720/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202089.988602:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202089.988608:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202089.988633:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187663232/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202089.988646:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187662720/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202089.988656:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202089.988660:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202089.988665:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187662720:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202089.988671:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187662848:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202089.988692:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202089.988704:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202089/real 1788202089] req@ffff8e828953a680 x1875062187662848/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202089.988714:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187662848/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202089.988722:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202089.988725:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202089.988728:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187662848:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202089.988749:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187662976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202089.988764:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202089.988773:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202089/real 1788202089] req@ffff8e8289538a80 x1875062187662976/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202089.988784:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187662976/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202089.988792:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202089.988794:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202089.988797:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187662976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202089.988803:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187663104:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202089.988818:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202089.988825:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202089/real 1788202089] req@ffff8e8289539500 x1875062187663104/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202089.988832:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187663104/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202089.988837:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202089.988839:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202089.988840:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187663104:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202089.988843:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187663232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202089.988851:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202089.988856:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202089/real 1788202089] req@ffff8e828953bb80 x1875062187663232/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202089.988862:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187663232/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202089.988867:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202089.988875:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202089.988877:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187663232:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202090.692477:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202091.716471:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202092.740433:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202093.764542:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202094.788490:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202095.108481:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202095.108493:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202095.108498:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202095.108510:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3550 00000100:00080000:1.0:1788202095.108522:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202095.108562:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202095.108606:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187663360/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202095.108636:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202095.108640:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202095.108642:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202095.108652:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3550 00000100:00080000:1.0:1788202095.108656:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202095.108661:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202095.108672:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202095.108684:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187663488/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202095.108690:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187663360:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202095.108699:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202095.108704:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202095.108705:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202095.108710:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3550 00000100:00080000:1.0:1788202095.108713:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202095.108716:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202095.108721:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202095.108733:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187663616/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202095.108743:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202095.108746:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202095.108748:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202095.108758:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3550 00000100:00080000:1.0:1788202095.108769:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202095.108773:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202095.108778:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788202095.108784:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202095.108791:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187663744/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202095.108802:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202095.108806:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202095.108808:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202095.108819:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3550 00000100:00080000:1.0:1788202095.108823:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202095.108828:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202095.108834:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202095.108839:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202095/real 1788202095] req@ffff8e828d722d80 x1875062187663360/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202095.108850:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187663872/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202095.108861:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187663360/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202095.108874:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202095.108878:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202095.108883:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187663360:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202095.108910:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187663488:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202095.108931:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202095.108943:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202095/real 1788202095] req@ffff8e828d721180 x1875062187663488/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202095.108954:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187663488/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202095.108961:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202095.108964:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202095.108968:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187663488:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202095.108974:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187663616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202095.108991:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202095.109000:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202095/real 1788202095] req@ffff8e828d720a80 x1875062187663616/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202095.109011:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187663616/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202095.109018:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202095.109021:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202095.109024:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187663616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202095.109030:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187663744:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202095.109047:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202095.109056:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202095/real 1788202095] req@ffff8e828d720380 x1875062187663744/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202095.109066:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187663744/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202095.109073:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202095.109075:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202095.109077:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187663744:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202095.109082:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187663872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202095.109095:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202095.109103:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202095/real 1788202095] req@ffff8e828d722300 x1875062187663872/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202095.109113:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187663872/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202095.109120:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202095.109123:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202095.109125:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187663872:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202095.812477:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202096.836522:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202097.860513:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202098.884479:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202099.908484:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202100.228485:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202100.228493:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202100.228496:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202100.228503:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3555 00000100:00080000:2.0:1788202100.228512:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202100.228517:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202100.228552:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187664000/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202100.228578:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202100.228580:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202100.228582:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202100.228584:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3555 00000100:00080000:2.0:1788202100.228587:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202100.228590:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202100.228598:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202100.228605:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187664000:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202100.228609:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187664128/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202100.228617:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202100.228619:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202100.228620:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202100.228623:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3555 00000100:00080000:2.0:1788202100.228625:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202100.228627:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202100.228630:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202100.228636:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187664256/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202100.228642:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202100.228645:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202100.228646:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202100.228650:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3555 00000100:00080000:2.0:1788202100.228654:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788202100.228662:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788202100.228667:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202100.228676:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202100.228689:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187664384/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202100.228697:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202100.228701:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1788202100.228702:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202100/real 1788202100] req@ffff8e82894e4e00 x1875062187664000/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202100.228703:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202100.228710:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3555 00000100:00080000:2.0:1788202100.228717:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202100.228720:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202100.228724:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187664000/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788202100.228725:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788202100.228736:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1788202100.228739:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187664512/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202100.228742:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202100.228748:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187664000:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202100.228770:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187664128:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202100.228790:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202100.228801:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202100/real 1788202100] req@ffff8e82894e7800 x1875062187664128/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202100.228813:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187664128/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202100.228821:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202100.228823:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202100.228826:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187664128:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202100.228832:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187664256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202100.228845:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202100.228852:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202100/real 1788202100] req@ffff8e82894e6300 x1875062187664256/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202100.228859:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187664256/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202100.228863:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202100.228865:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202100.228867:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187664256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202100.228870:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187664384:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202100.228880:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202100.228885:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202100/real 1788202100] req@ffff8e82894e4a80 x1875062187664384/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202100.228891:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187664384/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202100.228896:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202100.228897:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202100.228898:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187664384:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202100.228901:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187664512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202100.228910:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202100.228914:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202100/real 1788202100] req@ffff8e82894e6d80 x1875062187664512/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202100.228920:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187664512/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202100.228924:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202100.228926:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202100.228927:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187664512:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202100.932546:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202101.956492:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202102.980533:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202104.004524:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202105.028491:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202105.348465:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202105.348474:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202105.348479:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202105.348494:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3560 00000100:00080000:2.0:1788202105.348507:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202105.348516:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202105.348558:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187664640/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202105.348582:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202105.348587:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202105.348589:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202105.348594:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3560 00000100:00080000:2.0:1788202105.348599:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202105.348603:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202105.348618:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202105.348628:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187664640:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202105.348641:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187664768/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202105.348655:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202105.348659:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202105.348662:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202105.348668:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3560 00000100:00080000:2.0:1788202105.348673:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202105.348678:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202105.348685:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202105.348699:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187664896/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202105.348709:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202105.348713:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202105.348715:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202105.348721:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3560 00000100:00080000:2.0:1788202105.348726:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202105.348731:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202105.348737:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202105.348746:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187664768:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202105.348752:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187665024/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202105.348760:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202105.348763:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202105.348765:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202105.348768:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3560 00000100:00080000:2.0:1788202105.348771:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202105.348777:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202105.348780:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187664896:192.168.201.121@tcp:38:kworker.0 00000008:00100000:2.0:1788202105.348790:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202105.348804:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187665024:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202105.348811:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187665152/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202105.348860:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187665152:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202106.052498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202107.076557:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202108.100484:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788202108.420552:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788202108.420594:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788202108.420617:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202108.420655:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202108.420664:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202108.420671:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202108.420679:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788202108.420707:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202105/real 1788202108] req@ffff8e82894e6d80 x1875062187665152/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202108.420729:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187665152/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202108.420739:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202108.420744:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202108.420748:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187665152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202108.420755:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202105/real 1788202108] req@ffff8e82894e5f80 x1875062187665024/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202108.420765:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187665024/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202108.420771:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202108.420775:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202108.420778:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187665024:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202108.420784:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202105/real 1788202108] req@ffff8e82894e4380 x1875062187664768/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202108.420807:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187664768/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202108.420814:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202108.420818:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202108.420821:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187664768:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202108.420827:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202105/real 1788202108] req@ffff8e82894e5c00 x1875062187664896/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202108.420836:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187664896/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202108.420842:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202108.420845:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202108.420847:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187664896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202108.420853:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202105/real 1788202108] req@ffff8e82894e7100 x1875062187664640/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202108.420871:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187664640/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202108.420878:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202108.420881:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202108.420884:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187664640:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788202109.124493:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202110.148463:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202110.468439:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202110.468447:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202110.468451:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202110.468458:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3565 00000100:00080000:0.0:1788202110.468468:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202110.468473:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202110.468500:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187665280/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202110.468521:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202110.468524:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202110.468525:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202110.468528:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3565 00000100:00080000:0.0:1788202110.468531:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202110.468533:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202110.468540:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202110.468548:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187665408/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202110.468557:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202110.468561:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202110.468563:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202110.468566:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3565 00000100:00080000:0.0:1788202110.468570:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202110.468579:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187665280:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202110.468586:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202110.468593:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202110.468605:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187665536/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202110.468612:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202110.468616:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202110.468617:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202110.468621:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3565 00000100:00080000:0.0:1788202110.468644:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788202110.468646:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202110.468651:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202110.468658:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202110.468674:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187665664/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202110.468684:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202110.468688:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202110.468689:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202110.468694:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3565 00000100:00080000:0.0:1788202110.468697:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202110.468701:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202110/real 1788202110] req@ffff8e828793c000 x1875062187665280/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202110.468705:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202110.468713:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202110.468728:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187665792/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202110.468728:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187665280/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202110.468745:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202110.468752:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202110.468758:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187665280:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202110.468765:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187665408:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202110.468792:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202110.468804:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202110/real 1788202110] req@ffff8e828793f480 x1875062187665408/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202110.468815:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187665408/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202110.468823:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202110.468827:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202110.468830:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187665408:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202110.468851:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187665536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202110.468870:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202110.468880:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202110/real 1788202110] req@ffff8e828793dc00 x1875062187665536/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202110.468891:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187665536/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202110.468898:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202110.468902:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202110.468905:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187665536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202110.468910:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187665664:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202110.468926:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202110.468935:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202110/real 1788202110] req@ffff8e828793c700 x1875062187665664/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202110.468946:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187665664/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202110.468954:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202110.468957:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202110.468960:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187665664:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202110.468966:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187665792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202110.468981:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202110.468990:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202110/real 1788202110] req@ffff8e828793ca80 x1875062187665792/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202110.469008:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187665792/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202110.469016:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202110.469019:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202110.469022:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187665792:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202111.172440:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202112.196514:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202113.220442:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202114.244453:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202115.268452:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202115.588424:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202115.588431:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202115.588435:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202115.588443:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3570 00000100:00080000:2.0:1788202115.588455:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202115.588460:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202115.588490:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187665920/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202115.588511:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202115.588514:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202115.588515:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202115.588518:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3570 00000100:00080000:2.0:1788202115.588522:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202115.588525:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202115.588531:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202115.588538:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187666048/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202115.588544:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202115.588546:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202115.588547:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202115.588548:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3570 00000100:00080000:2.0:1788202115.588550:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202115.588552:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202115.588554:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202115.588559:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187666176/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202115.588564:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202115.588566:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202115.588567:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202115.588570:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3570 00000100:00080000:2.0:1788202115.588572:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202115.588574:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202115.588578:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202115.588582:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187665920:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202115.588585:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187666304/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202115.588598:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202115.588600:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202115.588601:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202115.588603:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3570 00000100:00080000:2.0:1788202115.588605:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202115.588607:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202115.588609:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202115.588617:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187666432/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202115.588643:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202115.588680:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202115/real 1788202115] req@ffff8e82894e4a80 x1875062187665920/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202115.588691:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187665920/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202115.588698:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202115.588700:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202115.588702:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187665920:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202115.588709:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187666048:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202115.588728:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202115.588740:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202115/real 1788202115] req@ffff8e82894e6300 x1875062187666048/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202115.588751:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187666048/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202115.588759:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202115.588762:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202115.588765:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187666048:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202115.588770:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187666176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202115.588787:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202115.588796:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202115/real 1788202115] req@ffff8e82894e7800 x1875062187666176/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202115.588807:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187666176/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202115.588815:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202115.588818:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202115.588821:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187666176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202115.588841:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187666304:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202115.588853:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202115.588859:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202115/real 1788202115] req@ffff8e82894e4e00 x1875062187666304/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202115.588865:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187666304/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202115.588874:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202115.588876:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202115.588878:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187666304:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202115.588881:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187666432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202115.588889:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202115.588893:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202115/real 1788202115] req@ffff8e82894e7480 x1875062187666432/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202115.588899:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187666432/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202115.588911:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202115.588913:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202115.588914:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187666432:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202116.292485:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202117.316520:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202118.340507:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202119.364476:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202120.388511:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202120.708463:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202120.708476:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202120.708480:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202120.708491:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3575 00000100:00080000:2.0:1788202120.708503:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202120.708511:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202120.708558:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187666560/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202120.708587:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202120.708591:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202120.708592:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202120.708595:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3575 00000100:00080000:2.0:1788202120.708598:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202120.708600:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202120.708611:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202120.708620:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187666688/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202120.708628:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202120.708631:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202120.708633:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202120.708635:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3575 00000100:00080000:2.0:1788202120.708637:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202120.708640:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202120.708643:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202120.708648:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187666816/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202120.708657:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788202120.708659:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187666560:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202120.708661:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202120.708666:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202120.708675:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3575 00000100:00080000:2.0:1788202120.708678:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202120.708682:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202120.708688:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202120.708703:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187666944/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202120.708714:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202120.708719:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202120.708722:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202120.708725:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3575 00000100:00080000:2.0:1788202120.708729:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202120.708732:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202120.708736:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202120.708746:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187667072/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202120.708758:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202120.708816:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202120/real 1788202120] req@ffff8e82894e6680 x1875062187666560/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202120.708837:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187666560/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202120.708851:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202120.708857:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202120.708862:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187666560:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202120.708872:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187666688:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202120.708899:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202120.708913:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202120/real 1788202120] req@ffff8e82894e6a00 x1875062187666688/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202120.708926:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187666688/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202120.708934:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202120.708938:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202120.708942:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187666688:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202120.708971:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187666816:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202120.709007:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202120.709021:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202120/real 1788202120] req@ffff8e82894e4000 x1875062187666816/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202120.709035:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187666816/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202120.709043:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202120.709048:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202120.709052:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187666816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202120.709060:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187666944:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202120.709099:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202120.709115:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202120/real 1788202120] req@ffff8e82894e5880 x1875062187666944/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202120.709128:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187666944/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202120.709137:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202120.709141:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202120.709144:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187666944:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202120.709151:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187667072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202120.709172:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202120.709185:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202120/real 1788202120] req@ffff8e82894e7480 x1875062187667072/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202120.709198:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187667072/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202120.709207:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202120.709210:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202120.709213:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187667072:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202121.412534:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202122.436573:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202123.460439:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202124.484500:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202125.508483:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202125.828432:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202125.828443:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202125.828447:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202125.828458:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3580 00000100:00080000:0.0:1788202125.828469:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202125.828476:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202125.828509:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187667200/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202125.828536:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202125.828539:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202125.828541:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202125.828545:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3580 00000100:00080000:0.0:1788202125.828547:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202125.828549:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202125.828558:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202125.828567:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187667328/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202125.828575:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202125.828577:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202125.828578:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202125.828581:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3580 00000100:00080000:0.0:1788202125.828583:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202125.828585:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202125.828588:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202125.828593:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187667200:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202125.828598:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187667456/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202125.828608:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202125.828611:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202125.828612:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202125.828614:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3580 00000100:00080000:0.0:1788202125.828616:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202125.828618:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202125.828621:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202125.828635:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187667584/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202125.828645:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202125.828648:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202125.828651:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202125.828655:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3580 00000100:00080000:0.0:1788202125.828659:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788202125.828666:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202125.828667:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202125.828675:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202125.828691:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187667712/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202125.828703:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202125/real 1788202125] req@ffff8e828793f100 x1875062187667200/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202125.828719:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187667200/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202125.828727:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202125.828729:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202125.828732:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187667200:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202125.828737:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187667328:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202125.828751:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202125.828764:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202125/real 1788202125] req@ffff8e828793ce00 x1875062187667328/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202125.828777:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187667328/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202125.828785:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202125.828789:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202125.828793:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187667328:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202125.828818:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187667456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202125.828841:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202125.828854:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202125/real 1788202125] req@ffff8e828793f800 x1875062187667456/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202125.828867:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187667456/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202125.828876:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202125.828879:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202125.828881:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187667456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202125.828885:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187667584:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202125.828900:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202125.828908:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202125/real 1788202125] req@ffff8e828793e680 x1875062187667584/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202125.828915:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187667584/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202125.828921:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202125.828923:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202125.828925:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187667584:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202125.828928:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187667712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202125.828937:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202125.828943:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202125/real 1788202125] req@ffff8e828793e300 x1875062187667712/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202125.828949:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187667712/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202125.828959:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202125.828961:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202125.828963:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187667712:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202126.532472:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202127.556503:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202128.580528:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202129.604490:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202130.628489:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202130.948373:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202130.948382:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202130.948386:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202130.948393:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3586 00000100:00080000:0.0:1788202130.948402:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202130.948409:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202130.948442:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187667840/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202130.948470:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202130.948475:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202130.948477:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202130.948481:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3586 00000100:00080000:0.0:1788202130.948485:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202130.948489:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202130.948498:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202130.948513:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187667968/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202130.948528:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202130.948530:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202130.948531:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202130.948535:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3586 00000100:00080000:0.0:1788202130.948538:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202130.948552:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202130.948552:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187667840:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788202130.948558:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202130.948569:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187668096/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202130.948577:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202130.948580:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202130.948581:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202130.948583:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3586 00000100:00080000:0.0:1788202130.948586:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202130.948588:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202130.948592:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202130.948604:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187668224/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202130.948613:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202130.948616:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202130.948618:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202130.948624:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3586 00000800:00000100:1.0:1788202130.948626:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202130.948632:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202130.948639:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202130.948644:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202130.948657:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187668352/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202130.948680:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202130/real 1788202130] req@ffff8e828793e300 x1875062187667840/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202130.948701:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187667840/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202130.948713:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202130.948717:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202130.948721:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187667840:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202130.948728:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187667968:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202130.948747:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202130.948757:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202130/real 1788202130] req@ffff8e828793e680 x1875062187667968/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202130.948765:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187667968/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202130.948771:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202130.948774:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202130.948785:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187667968:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202130.948804:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187668096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202130.948817:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202130.948824:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202130/real 1788202130] req@ffff8e828793f800 x1875062187668096/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202130.948832:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187668096/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202130.948837:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202130.948840:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202130.948842:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187668096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202130.948845:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187668224:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202130.948856:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202130.948862:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202130/real 1788202130] req@ffff8e828793ce00 x1875062187668224/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202130.948869:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187668224/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202130.948874:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202130.948875:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202130.948878:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187668224:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202130.948882:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187668352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202130.948891:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202130.948896:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202130/real 1788202130] req@ffff8e828793f100 x1875062187668352/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202130.948920:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187668352/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202130.948926:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202130.948928:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202130.948930:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187668352:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202131.652445:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202132.676542:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202133.700496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202134.724505:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202135.748511:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202136.068429:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202136.068439:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202136.068443:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202136.068452:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3591 00000100:00080000:2.0:1788202136.068462:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202136.068468:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202136.068500:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187668480/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202136.068534:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202136.068536:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202136.068538:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202136.068544:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3591 00000100:00080000:2.0:1788202136.068546:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202136.068549:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202136.068557:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202136.068565:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187668608/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202136.068580:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187668480:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202136.068580:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202136.068593:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202136.068594:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202136.068598:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3591 00000100:00080000:2.0:1788202136.068600:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202136.068603:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202136.068606:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202136.068616:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187668736/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202136.068623:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202136.068625:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202136.068626:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202136.068628:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3591 00000100:00080000:2.0:1788202136.068631:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202136.068634:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202136.068637:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788202136.068642:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202136.068647:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187668864/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202136.068656:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202136.068661:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202136.068664:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202136.068667:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3591 00000100:00080000:2.0:1788202136.068670:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202136.068673:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202136.068676:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202136.068688:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187668992/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202136.068696:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202136/real 1788202136] req@ffff8e82894e4e00 x1875062187668480/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202136.068718:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187668480/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202136.068730:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202136.068735:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202136.068740:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187668480:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202136.068761:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187668608:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202136.068780:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202136.068789:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202136/real 1788202136] req@ffff8e82894e7800 x1875062187668608/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202136.068798:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187668608/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202136.068804:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202136.068806:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202136.068808:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187668608:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202136.068828:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187668736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202136.068847:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202136.068860:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202136/real 1788202136] req@ffff8e82894e6300 x1875062187668736/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202136.068873:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187668736/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202136.068882:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202136.068894:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202136.068898:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187668736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202136.068903:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187668864:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202136.068921:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202136.068929:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202136/real 1788202136] req@ffff8e82894e4a80 x1875062187668864/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202136.068937:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187668864/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202136.068945:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202136.068948:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202136.068952:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187668864:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202136.068958:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187668992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202136.068978:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202136.068990:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202136/real 1788202136] req@ffff8e82894e7100 x1875062187668992/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202136.069003:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187668992/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202136.069008:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202136.069011:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202136.069013:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187668992:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202136.772536:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202137.796571:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202138.820493:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202139.844512:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202140.868472:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202141.188463:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202141.188476:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202141.188481:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202141.188492:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3596 00000100:00080000:0.0:1788202141.188505:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202141.188514:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202141.188556:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187669120/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202141.188580:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202141.188584:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202141.188585:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202141.188589:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3596 00000100:00080000:0.0:1788202141.188591:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202141.188595:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202141.188604:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202141.188617:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187669120:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202141.188619:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187669248/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202141.188631:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202141.188636:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202141.188637:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202141.188641:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3596 00000100:00080000:0.0:1788202141.188643:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202141.188646:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202141.188649:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202141.188658:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187669376/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202141.188665:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202141.188667:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202141.188669:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202141.188672:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3596 00000100:00080000:0.0:1788202141.188675:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202141.188679:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202141.188684:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202141.188692:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187669504/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202141.188698:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202141.188707:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202141.188711:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202141.188714:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202141.188719:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3596 00000100:00080000:0.0:1788202141.188737:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202141.188742:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202141.188750:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202141.188760:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202141/real 1788202141] req@ffff8e828793ca80 x1875062187669120/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202141.188767:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187669632/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202141.188784:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187669120/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202141.188799:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202141.188804:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202141.188810:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187669120:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202141.188839:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187669248:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202141.188863:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202141.188878:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202141/real 1788202141] req@ffff8e828793c700 x1875062187669248/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202141.188892:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187669248/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202141.188901:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202141.188905:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202141.188910:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187669248:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202141.188915:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187669376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202141.188933:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202141.188943:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202141/real 1788202141] req@ffff8e828793dc00 x1875062187669376/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202141.188952:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187669376/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202141.188960:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202141.188963:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202141.188967:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187669376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202141.188975:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187669504:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202141.188998:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202141.189010:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202141/real 1788202141] req@ffff8e828793f480 x1875062187669504/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202141.189019:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187669504/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202141.189025:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202141.189028:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202141.189030:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187669504:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202141.189034:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187669632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202141.189049:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202141.189058:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202141/real 1788202141] req@ffff8e828793c000 x1875062187669632/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202141.189065:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187669632/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202141.189071:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202141.189073:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202141.189075:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187669632:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202141.892449:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202142.916503:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202143.940553:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202144.964562:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202145.988545:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202146.308400:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202146.308409:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202146.308413:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202146.308421:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3601 00000100:00080000:0.0:1788202146.308429:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202146.308435:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202146.308467:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793df80 x1875062187669760/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202146.308487:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202146.308489:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202146.308491:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202146.308494:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3601 00000100:00080000:0.0:1788202146.308496:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202146.308498:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202146.308505:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202146.308512:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187669888/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202146.308518:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202146.308520:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202146.308522:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202146.308524:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3601 00000100:00080000:0.0:1788202146.308527:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202146.308529:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202146.308533:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202146.308539:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187670016/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202146.308545:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202146.308548:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202146.308550:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202146.308552:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3601 00000100:00080000:0.0:1788202146.308554:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202146.308557:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202146.308561:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202146.308566:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187670144/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202146.308571:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202146.308573:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202146.308575:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202146.308579:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3601 00000100:00080000:0.0:1788202146.308582:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202146.308586:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202146.308589:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202146.308602:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187670272/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202146.308605:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187669760:192.168.201.121@tcp:250:kworker.0 00000800:00000100:2.0:1788202146.308746:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202146.308820:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202146/real 1788202146] req@ffff8e828793df80 x1875062187669760/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202146.308843:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793df80 x1875062187669760/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202146.308857:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202146.308862:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202146.308866:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187669760:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202146.308875:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187669888:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202146.308902:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202146.308917:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202146/real 1788202146] req@ffff8e828793d500 x1875062187669888/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202146.308931:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187669888/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202146.308947:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202146.308951:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202146.308954:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187669888:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202146.308962:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187670016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202146.308985:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202146.308998:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202146/real 1788202146] req@ffff8e828793d880 x1875062187670016/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202146.309012:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187670016/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202146.309021:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202146.309025:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202146.309029:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187670016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202146.309036:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187670144:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202146.309059:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202146.309073:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202146/real 1788202146] req@ffff8e828793c380 x1875062187670144/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202146.309086:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187670144/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202146.309095:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202146.309099:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202146.309103:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187670144:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202146.309137:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187670272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202146.309160:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202146.309172:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202146/real 1788202146] req@ffff8e828793fb80 x1875062187670272/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202146.309185:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187670272/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202146.309193:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202146.309197:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202146.309200:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187670272:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202147.012575:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202148.036554:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202149.060520:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202150.084531:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202151.108500:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202151.428444:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202151.428452:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202151.428456:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202151.428464:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3606 00000100:00080000:1.0:1788202151.428473:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202151.428479:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202151.428511:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187670400/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202151.428539:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202151.428543:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202151.428558:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202151.428563:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3606 00000100:00080000:1.0:1788202151.428567:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202151.428572:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202151.428582:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202151.428598:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187670528/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202151.428609:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202151.428612:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202151.428615:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202151.428618:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3606 00000100:00080000:1.0:1788202151.428622:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202151.428625:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202151.428629:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202151.428639:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187670400:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202151.428643:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187670656/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202151.428662:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202151.428665:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202151.428666:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202151.428671:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3606 00000100:00080000:1.0:1788202151.428673:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202151.428677:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202151.428680:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202151.428689:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187670784/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202151.428695:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202151.428697:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202151.428699:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202151.428703:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3606 00000100:00080000:1.0:1788202151.428706:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202151.428708:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202151.428711:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202151.428717:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187670912/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202151.428745:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202151.428823:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202151/real 1788202151] req@ffff8e828d722300 x1875062187670400/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202151.428846:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187670400/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202151.428856:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202151.428859:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202151.428863:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187670400:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202151.428870:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187670528:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202151.428890:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202151.428900:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202151/real 1788202151] req@ffff8e828d720380 x1875062187670528/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202151.428908:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187670528/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202151.428914:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202151.428916:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202151.428918:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187670528:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202151.428943:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187670656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202151.428956:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202151.428962:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202151/real 1788202151] req@ffff8e828d720a80 x1875062187670656/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202151.428970:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187670656/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202151.428975:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202151.428977:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202151.428979:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187670656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202151.428982:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187670784:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202151.428994:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202151.429000:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202151/real 1788202151] req@ffff8e828d721180 x1875062187670784/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202151.429007:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187670784/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202151.429013:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202151.429015:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202151.429017:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187670784:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202151.429020:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187670912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202151.429030:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202151.429036:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202151/real 1788202151] req@ffff8e828d722d80 x1875062187670912/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202151.429042:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187670912/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202151.429047:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202151.429049:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202151.429051:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187670912:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202152.132574:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202153.156680:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202154.180586:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202155.204562:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202156.228521:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202156.548441:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202156.548450:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202156.548453:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202156.548475:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3611 00000100:00080000:1.0:1788202156.548483:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202156.548489:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202156.548520:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187671040/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202156.548545:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202156.548548:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202156.548550:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202156.548553:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3611 00000100:00080000:1.0:1788202156.548556:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202156.548558:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202156.548566:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202156.548575:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187671168/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202156.548582:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202156.548591:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202156.548592:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202156.548596:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3611 00000100:00080000:1.0:1788202156.548600:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788202156.548601:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187671040:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202156.548603:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202156.548610:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202156.548618:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187671296/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202156.548626:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202156.548628:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202156.548629:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202156.548632:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3611 00000100:00080000:1.0:1788202156.548634:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202156.548636:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202156.548646:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202156.548654:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187671424/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202156.548660:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202156.548663:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202156.548664:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202156.548666:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3611 00000100:00080000:1.0:1788202156.548668:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202156.548670:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202156.548674:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788202156.548676:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202156.548683:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187671552/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202156.548720:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202156/real 1788202156] req@ffff8e828d720e00 x1875062187671040/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202156.548735:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187671040/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202156.548742:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202156.548745:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202156.548748:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187671040:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202156.548754:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187671168:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202156.548771:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202156.548779:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202156/real 1788202156] req@ffff8e828d720000 x1875062187671168/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202156.548787:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187671168/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202156.548792:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202156.548794:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202156.548797:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187671168:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202156.548818:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187671296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202156.548838:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202156.548850:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202156/real 1788202156] req@ffff8e828d721500 x1875062187671296/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202156.548878:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187671296/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202156.548887:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202156.548891:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202156.548895:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187671296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202156.548902:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187671424:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202156.548924:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202156.548936:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202156/real 1788202156] req@ffff8e828d723800 x1875062187671424/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202156.548950:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187671424/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202156.548958:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202156.548961:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202156.548964:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187671424:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202156.548971:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187671552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202156.548993:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202156.549004:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202156/real 1788202156] req@ffff8e828d722680 x1875062187671552/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202156.549015:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187671552/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202156.549020:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202156.549022:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202156.549024:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187671552:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202157.252533:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202158.276547:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202159.300494:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202160.324522:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202161.348527:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202161.668472:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202161.668482:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202161.668487:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202161.668499:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3616 00000100:00080000:3.0:1788202161.668517:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202161.668526:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202161.668569:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187671680/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202161.668597:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202161.668600:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202161.668602:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202161.668605:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3616 00000100:00080000:3.0:1788202161.668608:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202161.668611:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202161.668621:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202161.668635:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187671808/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202161.668646:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202161.668649:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202161.668652:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202161.668656:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187671680:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202161.668656:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3616 00000100:00080000:3.0:1788202161.668660:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202161.668666:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202161.668670:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202161.668676:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187671936/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202161.668683:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202161.668685:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202161.668687:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202161.668689:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3616 00000100:00080000:3.0:1788202161.668691:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202161.668693:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202161.668696:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202161.668707:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187672064/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202161.668713:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202161.668716:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202161.668717:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202161.668720:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3616 00000100:00080000:3.0:1788202161.668729:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202161.668731:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202161.668734:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202161.668742:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187672192/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202161.668750:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202161.668807:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202161/real 1788202161] req@ffff8e8289539f80 x1875062187671680/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202161.668827:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187671680/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202161.668841:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202161.668844:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202161.668847:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187671680:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202161.668870:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187671808:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202161.668890:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202161.668900:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202161/real 1788202161] req@ffff8e8289538380 x1875062187671808/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202161.668908:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187671808/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202161.668914:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202161.668916:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202161.668918:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187671808:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202161.668922:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187671936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202161.668933:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202161.668940:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202161/real 1788202161] req@ffff8e828953b100 x1875062187671936/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202161.668947:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187671936/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202161.668952:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202161.668954:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202161.668956:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187671936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202161.668960:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187672064:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202161.668971:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202161.668977:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202161/real 1788202161] req@ffff8e828953bb80 x1875062187672064/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202161.668984:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187672064/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202161.668990:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202161.668991:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202161.668993:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187672064:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202161.668997:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187672192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202161.669006:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202161.669012:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202161/real 1788202161] req@ffff8e8289539500 x1875062187672192/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202161.669019:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187672192/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202161.669024:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202161.669026:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202161.669028:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187672192:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202162.372489:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202163.396487:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202164.420517:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202165.444481:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202166.468457:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202166.788410:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202166.788420:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202166.788424:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202166.788434:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3621 00000100:00080000:3.0:1788202166.788444:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202166.788451:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202166.788494:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187672320/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202166.788520:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202166.788525:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202166.788526:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202166.788531:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3621 00000100:00080000:3.0:1788202166.788535:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202166.788546:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202166.788557:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202166.788572:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187672320:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202166.788575:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187672448/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202166.788590:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202166.788594:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202166.788596:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202166.788601:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3621 00000100:00080000:3.0:1788202166.788605:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202166.788610:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202166.788616:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202166.788638:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187672576/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202166.788645:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788202166.788655:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202166.788659:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202166.788662:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202166.788668:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3621 00000100:00080000:3.0:1788202166.788672:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202166.788676:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202166.788681:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202166.788698:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202166/real 1788202166] req@ffff8e8289538a80 x1875062187672320/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202166.788699:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187672704/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202166.788726:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202166.788731:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202166.788734:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202166.788740:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3621 00000100:00100000:2.0:1788202166.788741:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187672320/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202166.788744:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202166.788752:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202166.788761:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788202166.788763:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202166.788770:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202166.788774:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187672320:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202166.788778:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187672832/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202166.788799:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187672448:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202166.788820:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202166.788830:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202166/real 1788202166] req@ffff8e828953a680 x1875062187672448/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202166.788838:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187672448/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202166.788844:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202166.788847:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202166.788850:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187672448:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202166.788855:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187672576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202166.788868:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202166.788875:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202166/real 1788202166] req@ffff8e8289539880 x1875062187672576/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202166.788883:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187672576/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202166.788893:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202166.788895:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202166.788897:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187672576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202166.788900:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187672704:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202166.788911:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202166.788916:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202166/real 1788202166] req@ffff8e8289539500 x1875062187672704/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202166.788923:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187672704/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202166.788928:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202166.788931:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202166.788932:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187672704:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202166.788936:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187672832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202166.788945:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202166.788950:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202166/real 1788202166] req@ffff8e828953bb80 x1875062187672832/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202166.788956:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187672832/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202166.788961:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202166.788963:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202166.788965:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187672832:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202167.492483:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202168.516444:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202169.540449:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202170.564457:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202171.588480:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202171.908391:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202171.908406:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202171.908411:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202171.908421:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3626 00000100:00080000:3.0:1788202171.908433:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202171.908440:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202171.908476:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187672960/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202171.908504:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202171.908508:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202171.908510:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202171.908515:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3626 00000100:00080000:3.0:1788202171.908519:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202171.908528:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202171.908538:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202171.908551:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187672960:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202171.908556:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187673088/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202171.908571:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202171.908575:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202171.908577:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202171.908582:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3626 00000100:00080000:3.0:1788202171.908586:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202171.908590:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202171.908595:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202171.908608:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187673216/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202171.908618:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202171.908622:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202171.908624:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202171.908628:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3626 00000100:00080000:3.0:1788202171.908632:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202171.908638:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202171.908643:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202171.908653:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187673088:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202171.908659:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187673344/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202171.908673:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202171.908677:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202171.908680:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202171.908686:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3626 00000100:00100000:2.0:1788202171.908687:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187673216:192.168.201.121@tcp:38:kworker.0 00000100:00080000:3.0:1788202171.908693:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202171.908700:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202171.908705:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202171.908718:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187673472/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202171.908729:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187673344:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202171.908757:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187673472:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202172.612491:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202173.636477:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202174.660549:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788202174.980550:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:1.0:1788202174.980597:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788202174.980615:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788202174.980667:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788202174.980676:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788202174.980686:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788202174.980695:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:2.0:1788202174.980717:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202171/real 1788202174] req@ffff8e8289539500 x1875062187673472/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202174.980745:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187673472/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202174.980759:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202174.980764:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202174.980769:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187673472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202174.980777:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202171/real 1788202174] req@ffff8e828953bb80 x1875062187673344/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202174.980792:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187673344/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202174.980800:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202174.980803:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202174.980807:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187673344:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202174.980815:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202171/real 1788202174] req@ffff8e8289538380 x1875062187673088/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202174.980829:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187673088/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202174.980838:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202174.980842:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202174.980846:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187673088:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202174.980853:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202171/real 1788202174] req@ffff8e8289539f80 x1875062187673216/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202174.980864:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187673216/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202174.980872:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202174.980875:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202174.980879:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187673216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202174.980885:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202171/real 1788202174] req@ffff8e828953b100 x1875062187672960/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202174.980897:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187672960/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202174.980905:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202174.980907:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202174.980911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187672960:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788202175.684465:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202176.708487:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202177.028388:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202177.028397:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202177.028400:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202177.028408:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3632 00000100:00080000:3.0:1788202177.028417:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202177.028423:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202177.028457:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187673600/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202177.028487:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202177.028490:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202177.028492:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202177.028495:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3632 00000100:00080000:3.0:1788202177.028499:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202177.028503:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202177.028512:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202177.028529:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187673728/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202177.028542:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788202177.028546:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187673600:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202177.028554:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202177.028558:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202177.028561:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3632 00000100:00080000:3.0:1788202177.028564:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202177.028568:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202177.028572:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202177.028582:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187673856/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202177.028591:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202177.028604:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202177.028606:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202177.028618:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3632 00000100:00080000:3.0:1788202177.028622:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202177.028627:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202177.028632:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788202177.028637:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202177.028651:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187673984/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202177.028661:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202177.028664:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202177.028667:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202177.028671:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3632 00000100:00080000:3.0:1788202177.028676:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202177.028680:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202177.028687:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202177.028700:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187674112/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202177.028702:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202177/real 1788202177] req@ffff8e8289539880 x1875062187673600/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202177.028736:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187673600/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202177.028748:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202177.028753:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202177.028758:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187673600:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202177.028783:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187673728:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202177.028809:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202177.028823:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202177/real 1788202177] req@ffff8e828953a680 x1875062187673728/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202177.028839:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187673728/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202177.028848:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202177.028853:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202177.028857:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187673728:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202177.028864:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187673856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202177.028892:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202177.028908:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202177/real 1788202177] req@ffff8e8289538a80 x1875062187673856/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202177.028922:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187673856/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202177.028932:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202177.028936:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202177.028939:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187673856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202177.028946:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187673984:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202177.028983:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202177.028998:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202177/real 1788202177] req@ffff8e828953b100 x1875062187673984/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202177.029013:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187673984/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202177.029021:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202177.029024:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202177.029029:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187673984:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202177.029037:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187674112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202177.029059:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202177.029072:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202177/real 1788202177] req@ffff8e8289539f80 x1875062187674112/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202177.029084:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187674112/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202177.029136:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202177.029141:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202177.029144:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187674112:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202177.732494:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202178.756450:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202179.780466:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202180.804510:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202181.828470:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202182.148478:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202182.148491:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202182.148501:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202182.148513:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3637 00000100:00080000:3.0:1788202182.148526:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202182.148534:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202182.148582:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187674240/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202182.148613:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202182.148617:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202182.148619:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202182.148626:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3637 00000100:00080000:3.0:1788202182.148630:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202182.148634:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202182.148645:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202182.148661:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187674368/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202182.148665:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187674240:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202182.148678:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202182.148682:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202182.148684:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202182.148689:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3637 00000100:00080000:3.0:1788202182.148693:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202182.148697:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202182.148702:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202182.148712:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187674496/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202182.148721:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202182.148725:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202182.148727:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202182.148731:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3637 00000100:00080000:3.0:1788202182.148737:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202182.148741:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202182.148747:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788202182.148758:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202182.148759:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187674624/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202182.148769:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202182.148778:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202182.148781:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202182.148787:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3637 00000100:00080000:3.0:1788202182.148790:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202182.148794:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202182.148799:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202182.148812:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187674752/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202182.148828:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202182/real 1788202182] req@ffff8e8289538380 x1875062187674240/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202182.148851:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187674240/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202182.148868:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202182.148873:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202182.148878:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187674240:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202182.148906:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187674368:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202182.148929:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202182.148942:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202182/real 1788202182] req@ffff8e828953bb80 x1875062187674368/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202182.148953:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187674368/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202182.148961:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202182.148964:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202182.148967:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187674368:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202182.148974:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187674496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202182.148992:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202182.149001:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202182/real 1788202182] req@ffff8e8289539500 x1875062187674496/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202182.149023:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187674496/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202182.149031:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202182.149034:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202182.149037:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187674496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202182.149043:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187674624:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202182.149061:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202182.149071:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202182/real 1788202182] req@ffff8e8289539f80 x1875062187674624/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202182.149081:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187674624/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202182.149088:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202182.149090:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202182.149093:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187674624:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202182.149098:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187674752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202182.149114:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202182.149124:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202182/real 1788202182] req@ffff8e828953b100 x1875062187674752/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202182.149136:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187674752/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202182.149144:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202182.149146:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202182.149149:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187674752:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202182.852457:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202183.876454:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202184.900495:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202185.924563:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202186.948489:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202187.268397:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202187.268404:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202187.268408:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202187.268416:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3642 00000100:00080000:2.0:1788202187.268424:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202187.268429:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202187.268456:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187674880/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202187.268478:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202187.268480:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202187.268482:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202187.268484:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3642 00000100:00080000:2.0:1788202187.268489:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202187.268492:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202187.268498:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202187.268506:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187675008/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202187.268521:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187674880:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202187.268524:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202187.268529:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202187.268531:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202187.268536:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3642 00000100:00080000:2.0:1788202187.268540:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202187.268544:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202187.268549:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202187.268562:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187675136/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202187.268571:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202187.268573:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202187.268575:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202187.268578:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3642 00000100:00080000:2.0:1788202187.268580:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202187.268583:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202187.268586:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788202187.268588:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202187.268598:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187675264/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202187.268608:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202187.268611:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202187.268613:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202187.268616:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3642 00000100:00080000:2.0:1788202187.268621:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202187.268625:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202187.268629:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202187.268633:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202187/real 1788202187] req@ffff8e82894e5c00 x1875062187674880/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202187.268643:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187675392/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202187.268658:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187674880/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202187.268670:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202187.268674:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202187.268677:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187674880:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202187.268699:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187675008:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202187.268721:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202187.268732:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202187/real 1788202187] req@ffff8e82894e4380 x1875062187675008/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202187.268744:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187675008/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202187.268752:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202187.268755:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202187.268758:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187675008:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202187.268763:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187675136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202187.268778:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202187.268787:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202187/real 1788202187] req@ffff8e82894e5f80 x1875062187675136/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202187.268798:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187675136/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202187.268805:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202187.268808:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202187.268811:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187675136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202187.268816:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187675264:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202187.268830:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202187.268837:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202187/real 1788202187] req@ffff8e82894e6d80 x1875062187675264/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202187.268843:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187675264/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202187.268847:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202187.268850:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202187.268853:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187675264:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202187.268859:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187675392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202187.268876:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202187.268886:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202187/real 1788202187] req@ffff8e82894e7100 x1875062187675392/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202187.268897:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187675392/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202187.268904:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202187.268906:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202187.268909:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187675392:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202187.972455:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202188.996489:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202190.020432:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202191.044453:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202192.068487:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202192.388426:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202192.388437:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202192.388441:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202192.388452:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3647 00000100:00080000:3.0:1788202192.388463:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202192.388470:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202192.388515:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187675520/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202192.388537:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202192.388540:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202192.388541:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202192.388544:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3647 00000100:00080000:3.0:1788202192.388546:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202192.388548:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202192.388557:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202192.388567:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187675648/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202192.388573:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202192.388576:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202192.388577:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202192.388579:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3647 00000100:00080000:3.0:1788202192.388582:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202192.388585:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202192.388589:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202192.388600:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187675776/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202192.388611:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202192.388617:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202192.388621:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788202192.388622:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187675520:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202192.388632:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3647 00000100:00080000:3.0:1788202192.388637:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202192.388640:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202192.388645:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202192.388654:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187675904/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202192.388661:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202192.388664:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202192.388665:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202192.388669:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3647 00000100:00080000:3.0:1788202192.388674:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202192.388676:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202192.388680:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202192.388689:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187676032/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202192.388693:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202192.388740:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202192/real 1788202192] req@ffff8e8289538a80 x1875062187675520/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202192.388752:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187675520/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202192.388759:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202192.388762:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202192.388765:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187675520:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202192.388770:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187675648:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202192.388784:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202192.388791:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202192/real 1788202192] req@ffff8e828953a680 x1875062187675648/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202192.388797:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187675648/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202192.388801:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202192.388803:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202192.388805:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187675648:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202192.388822:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187675776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202192.388832:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202192.388853:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202192/real 1788202192] req@ffff8e8289539880 x1875062187675776/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202192.388861:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187675776/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202192.388865:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202192.388868:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202192.388869:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187675776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202192.388873:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187675904:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202192.388886:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202192.388892:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202192/real 1788202192] req@ffff8e828953b100 x1875062187675904/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202192.388898:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187675904/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202192.388903:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202192.388904:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202192.388906:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187675904:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202192.388909:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187676032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202192.388918:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202192.388923:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202192/real 1788202192] req@ffff8e8289539f80 x1875062187676032/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202192.388929:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187676032/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202192.388934:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202192.388935:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202192.388937:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187676032:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202193.092528:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202194.116477:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202195.140471:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202196.164465:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202197.188493:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202197.508413:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202197.508423:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202197.508428:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202197.508440:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3652 00000100:00080000:3.0:1788202197.508450:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202197.508457:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202197.508501:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187676160/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202197.508536:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202197.508546:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202197.508548:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202197.508553:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3652 00000100:00080000:3.0:1788202197.508557:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202197.508561:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202197.508570:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202197.508580:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187676160:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202197.508585:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187676288/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202197.508593:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202197.508596:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202197.508597:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202197.508600:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3652 00000100:00080000:3.0:1788202197.508602:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202197.508604:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202197.508607:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202197.508614:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187676416/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202197.508624:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202197.508626:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202197.508627:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202197.508629:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3652 00000100:00080000:3.0:1788202197.508631:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202197.508634:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202197.508636:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202197.508643:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187676544/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202197.508647:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202197.508650:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202197.508651:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202197.508654:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3652 00000800:00000100:1.0:1788202197.508657:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788202197.508657:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202197.508662:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202197.508665:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202197.508672:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187676672/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202197.508704:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202197/real 1788202197] req@ffff8e8289539500 x1875062187676160/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202197.508718:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187676160/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202197.508725:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202197.508728:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202197.508731:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187676160:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202197.508764:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187676288:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202197.508783:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202197.508796:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202197/real 1788202197] req@ffff8e828953bb80 x1875062187676288/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202197.508808:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187676288/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202197.508816:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202197.508819:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202197.508822:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187676288:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202197.508828:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187676416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202197.508846:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202197.508854:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202197/real 1788202197] req@ffff8e8289538380 x1875062187676416/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202197.508861:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187676416/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202197.508870:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202197.508873:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202197.508875:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187676416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202197.508878:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187676544:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202197.508889:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202197.508895:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202197/real 1788202197] req@ffff8e8289539f80 x1875062187676544/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202197.508901:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187676544/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202197.508905:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202197.508907:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202197.508909:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187676544:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202197.508912:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187676672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202197.508921:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202197.508925:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202197/real 1788202197] req@ffff8e828953b100 x1875062187676672/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202197.508932:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187676672/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202197.508936:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202197.508937:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202197.508940:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187676672:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202198.212504:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202199.236445:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202200.260455:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202201.284432:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202202.308460:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202202.628411:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202202.628419:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202202.628422:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202202.628429:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3657 00000100:00080000:1.0:1788202202.628437:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202202.628442:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202202.628473:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187676800/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202202.628499:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202202.628502:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202202.628503:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202202.628508:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3657 00000100:00080000:1.0:1788202202.628511:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202202.628515:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202202.628524:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202202.628538:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187676928/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202202.628541:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187676800:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202202.628555:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202202.628558:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202202.628560:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202202.628565:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3657 00000100:00080000:1.0:1788202202.628568:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202202.628572:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202202.628576:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202202.628587:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187677056/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202202.628595:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202202.628598:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202202.628600:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202202.628605:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3657 00000800:00000100:2.0:1788202202.628606:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788202202.628610:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202202.628615:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202202.628620:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202202.628630:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187677184/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202202.628638:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788202202.628642:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202202/real 1788202202] req@ffff8e828d722680 x1875062187676800/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202202.628643:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202202.628646:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202202.628653:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3657 00000100:00080000:1.0:1788202202.628669:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202202.628672:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202202.628675:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202202.628684:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187677312/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202202.628695:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187676800/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202202.628706:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202202.628709:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202202.628712:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187676800:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202202.628729:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187676928:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202202.628745:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202202.628753:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202202/real 1788202202] req@ffff8e828d723800 x1875062187676928/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202202.628761:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187676928/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202202.628765:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202202.628767:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202202.628770:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187676928:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202202.628773:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187677056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202202.628782:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202202.628787:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202202/real 1788202202] req@ffff8e828d721500 x1875062187677056/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202202.628793:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187677056/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202202.628800:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202202.628803:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202202.628806:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187677056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202202.628811:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187677184:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202202.628829:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202202.628847:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202202/real 1788202202] req@ffff8e828d720000 x1875062187677184/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202202.628855:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187677184/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202202.628859:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202202.628861:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202202.628863:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187677184:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202202.628866:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187677312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202202.628878:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202202.628884:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202202/real 1788202202] req@ffff8e828d720e00 x1875062187677312/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202202.628891:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187677312/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202202.628895:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202202.628896:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202202.628898:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187677312:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202203.332469:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202204.356554:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202205.380498:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202206.404541:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202207.428491:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202207.748415:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202207.748423:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202207.748426:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202207.748434:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3662 00000100:00080000:3.0:1788202207.748443:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202207.748449:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202207.748478:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187677440/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202207.748499:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202207.748501:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202207.748502:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202207.748508:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3662 00000100:00080000:3.0:1788202207.748511:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202207.748513:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202207.748520:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202207.748528:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187677568/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202207.748535:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202207.748538:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202207.748539:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202207.748542:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187677440:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202207.748544:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3662 00000100:00080000:3.0:1788202207.748550:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202207.748555:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202207.748560:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202207.748603:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187677696/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202207.748611:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788202207.748613:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788202207.748616:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202207.748618:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202207.748622:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3662 00000100:00080000:3.0:1788202207.748627:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202207.748632:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202207.748638:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202207.748651:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187677824/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202207.748661:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202207.748665:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202207.748666:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202207.748670:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202207/real 1788202207] req@ffff8e8289539880 x1875062187677440/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202207.748673:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3662 00000100:00080000:3.0:1788202207.748679:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202207.748687:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202207.748693:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202207.748696:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187677440/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202207.748706:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187677952/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202207.748709:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202207.748718:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202207.748723:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187677440:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202207.748750:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187677568:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202207.748770:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202207.748778:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202207/real 1788202207] req@ffff8e828953a680 x1875062187677568/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202207.748785:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187677568/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202207.748794:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202207.748796:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202207.748798:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187677568:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202207.748801:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187677696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202207.748811:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202207.748816:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202207/real 1788202207] req@ffff8e8289538a80 x1875062187677696/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202207.748822:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187677696/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202207.748827:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202207.748828:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202207.748830:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187677696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202207.748832:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187677824:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202207.748841:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202207.748846:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202207/real 1788202207] req@ffff8e828953b100 x1875062187677824/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202207.748852:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187677824/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202207.748856:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202207.748858:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202207.748860:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187677824:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202207.748863:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187677952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202207.748871:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202207.748876:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202207/real 1788202207] req@ffff8e8289539f80 x1875062187677952/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202207.748882:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187677952/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202207.748886:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202207.748887:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202207.748889:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187677952:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202208.452468:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202209.476454:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202210.500559:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202211.524466:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202212.548517:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202212.868491:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202212.868505:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202212.868511:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202212.868524:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3667 00000100:00080000:2.0:1788202212.868537:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202212.868546:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202212.868592:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187678080/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202212.868645:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202212.868651:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202212.868653:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202212.868658:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3667 00000100:00080000:2.0:1788202212.868663:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202212.868668:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202212.868680:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202212.868697:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187678208/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202212.868708:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202212.868712:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202212.868715:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202212.868719:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3667 00000100:00080000:2.0:1788202212.868723:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202212.868729:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202212.868735:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202212.868749:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187678336/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202212.868759:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202212.868762:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202212.868764:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202212.868769:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3667 00000100:00080000:2.0:1788202212.868773:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202212.868778:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202212.868783:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202212.868794:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187678464/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202212.868804:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202212.868808:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202212.868810:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202212.868813:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3667 00000100:00080000:2.0:1788202212.868817:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202212.868820:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202212.868825:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202212.868836:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187678592/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202212.869112:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187678080:192.168.201.121@tcp:250:kworker.0 00000800:00000100:3.0:1788202212.869227:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202212.869281:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202212/real 1788202212] req@ffff8e82894e4a80 x1875062187678080/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202212.869300:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187678080/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202212.869346:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202212.869352:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202212.869357:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187678080:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202212.869366:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187678208:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202212.869391:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202212.869407:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202212/real 1788202212] req@ffff8e82894e6300 x1875062187678208/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202212.869419:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187678208/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202212.869428:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202212.869431:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202212.869435:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187678208:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202212.869441:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187678336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202212.869459:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202212.869470:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202212/real 1788202212] req@ffff8e82894e7800 x1875062187678336/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202212.869497:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187678336/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202212.869506:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202212.869510:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202212.869514:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187678336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202212.869522:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187678464:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202212.869542:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202212.869553:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202212/real 1788202212] req@ffff8e82894e4e00 x1875062187678464/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202212.869565:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187678464/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202212.869573:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202212.869577:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202212.869580:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187678464:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202212.869586:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187678592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202212.869604:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202212.869614:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202212/real 1788202212] req@ffff8e82894e7480 x1875062187678592/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202212.869625:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187678592/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202212.869634:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202212.869637:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202212.869640:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187678592:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202213.572529:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202214.596529:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202215.620520:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202216.644480:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202217.668538:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202217.988432:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202217.988443:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202217.988448:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202217.988459:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3673 00000100:00080000:3.0:1788202217.988470:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202217.988477:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202217.988518:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187678720/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202217.988552:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202217.988556:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202217.988558:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202217.988564:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3673 00000100:00080000:3.0:1788202217.988568:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202217.988572:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202217.988584:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202217.988600:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187678848/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202217.988612:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202217.988617:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202217.988619:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202217.988624:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3673 00000100:00080000:3.0:1788202217.988630:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788202217.988632:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187678720:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202217.988638:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202217.988646:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202217.988661:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187678976/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202217.988681:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202217.988685:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202217.988687:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202217.988692:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3673 00000100:00080000:3.0:1788202217.988696:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202217.988700:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:2.0:1788202217.988706:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:3.0:1788202217.988706:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202217.988720:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187679104/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202217.988728:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202217.988731:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202217.988732:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202217.988735:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3673 00000100:00080000:3.0:1788202217.988738:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202217.988740:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202217.988743:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202217.988751:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187679232/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202217.988776:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202217/real 1788202217] req@ffff8e8289538380 x1875062187678720/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202217.988800:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187678720/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202217.988810:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202217.988813:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202217.988816:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187678720:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202217.988823:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187678848:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202217.988849:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202217.988864:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202217/real 1788202217] req@ffff8e828953bb80 x1875062187678848/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202217.988878:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187678848/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202217.988888:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202217.988891:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202217.988895:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187678848:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202217.988922:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187678976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202217.988939:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202217.988947:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202217/real 1788202217] req@ffff8e8289539500 x1875062187678976/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202217.988955:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187678976/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202217.988961:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202217.988964:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202217.988966:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187678976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202217.988969:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187679104:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202217.988981:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202217.988987:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202217/real 1788202217] req@ffff8e8289539f80 x1875062187679104/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202217.988994:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187679104/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202217.988999:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202217.989001:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202217.989004:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187679104:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202217.989007:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187679232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202217.989017:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202217.989022:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202217/real 1788202217] req@ffff8e828953b100 x1875062187679232/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202217.989029:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187679232/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202217.989033:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202217.989035:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202217.989037:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187679232:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202218.692486:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202219.716529:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202220.740461:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202221.764458:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202222.788498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202223.108393:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202223.108401:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202223.108405:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202223.108412:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3678 00000100:00080000:3.0:1788202223.108422:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202223.108427:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202223.108455:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187679360/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202223.108483:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202223.108487:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202223.108489:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202223.108493:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3678 00000100:00080000:3.0:1788202223.108496:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202223.108500:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202223.108519:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202223.108535:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187679488/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202223.108543:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187679360:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202223.108549:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202223.108554:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202223.108556:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202223.108560:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3678 00000100:00080000:3.0:1788202223.108564:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202223.108569:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202223.108575:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202223.108586:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187679616/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202223.108596:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788202223.108598:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788202223.108603:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202223.108606:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202223.108609:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3678 00000100:00080000:3.0:1788202223.108613:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202223.108616:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202223.108620:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202223.108628:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187679744/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202223.108634:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202223.108636:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202223.108637:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202223.108639:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3678 00000100:00080000:3.0:1788202223.108640:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202223.108642:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202223.108645:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202223.108648:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202223/real 1788202223] req@ffff8e8289538a80 x1875062187679360/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202223.108654:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187679872/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202223.108667:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187679360/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202223.108678:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202223.108682:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202223.108685:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187679360:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202223.108705:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187679488:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202223.108718:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202223.108726:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202223/real 1788202223] req@ffff8e828953a680 x1875062187679488/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202223.108733:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187679488/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202223.108738:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202223.108739:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202223.108741:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187679488:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202223.108744:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187679616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202223.108753:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202223.108758:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202223/real 1788202223] req@ffff8e8289539880 x1875062187679616/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202223.108764:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187679616/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202223.108768:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202223.108769:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202223.108771:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187679616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202223.108774:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187679744:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202223.108782:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202223.108786:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202223/real 1788202223] req@ffff8e828953b100 x1875062187679744/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202223.108793:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187679744/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202223.108797:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202223.108799:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202223.108801:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187679744:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202223.108803:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187679872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202223.108811:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202223.108816:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202223/real 1788202223] req@ffff8e8289539f80 x1875062187679872/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202223.108821:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187679872/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202223.108825:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202223.108827:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202223.108829:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187679872:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202223.812451:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202224.836425:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202225.860482:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202226.884466:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202227.908453:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202228.228391:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202228.228401:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202228.228405:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202228.228414:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3683 00000100:00080000:3.0:1788202228.228424:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202228.228430:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202228.228462:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187680000/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202228.228489:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202228.228492:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202228.228494:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202228.228498:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3683 00000100:00080000:3.0:1788202228.228502:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202228.228506:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202228.228515:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202228.228531:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187680000:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202228.228532:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187680128/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202228.228547:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202228.228551:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202228.228552:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202228.228556:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3683 00000100:00080000:3.0:1788202228.228559:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202228.228563:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202228.228567:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202228.228578:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187680256/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202228.228586:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202228.228590:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1788202228.228594:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788202228.228595:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202228.228601:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3683 00000100:00080000:3.0:1788202228.228605:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202228.228609:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202228.228613:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202228.228623:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187680384/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202228.228632:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202228.228635:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202228.228637:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202228.228641:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202228/real 1788202228] req@ffff8e8289539500 x1875062187680000/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202228.228649:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3683 00000100:00080000:3.0:1788202228.228655:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202228.228660:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202228.228664:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202228.228665:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187680000/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202228.228679:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187680512/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202228.228680:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202228.228689:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202228.228703:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187680000:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202228.228726:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187680128:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202228.228748:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202228.228760:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202228/real 1788202228] req@ffff8e828953bb80 x1875062187680128/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202228.228773:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187680128/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202228.228781:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202228.228783:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202228.228787:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187680128:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202228.228792:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187680256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202228.228809:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202228.228819:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202228/real 1788202228] req@ffff8e8289538380 x1875062187680256/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202228.228841:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187680256/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202228.228849:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202228.228853:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202228.228856:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187680256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202228.228862:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187680384:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202228.228880:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202228.228890:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202228/real 1788202228] req@ffff8e8289539f80 x1875062187680384/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202228.228902:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187680384/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202228.228909:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202228.228912:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202228.228916:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187680384:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202228.228921:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187680512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202228.228937:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202228.228945:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202228/real 1788202228] req@ffff8e828953b100 x1875062187680512/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202228.228956:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187680512/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202228.228964:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202228.228967:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202228.228970:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187680512:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202228.932457:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202229.956456:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202230.980468:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202232.004450:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202233.028465:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202233.348427:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202233.348436:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202233.348440:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202233.348449:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3688 00000100:00080000:0.0:1788202233.348457:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202233.348463:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202233.348497:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187680640/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202233.348521:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202233.348525:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202233.348527:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202233.348531:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3688 00000100:00080000:0.0:1788202233.348534:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202233.348539:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202233.348547:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202233.348555:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187680640:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202233.348561:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187680768/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202233.348573:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202233.348576:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202233.348578:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202233.348581:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3688 00000100:00080000:0.0:1788202233.348584:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202233.348588:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202233.348592:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202233.348601:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187680896/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202233.348609:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202233.348614:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202233.348617:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202233.348619:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202233.348623:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3688 00000100:00080000:0.0:1788202233.348627:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202233.348629:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202233.348633:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202233.348643:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187681024/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202233.348651:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202233.348654:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202233.348657:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202233.348658:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202233/real 1788202233] req@ffff8e828793fb80 x1875062187680640/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202233.348664:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3688 00000100:00080000:0.0:1788202233.348670:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202233.348677:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202233.348682:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202233.348686:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187680640/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202233.348698:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793df80 x1875062187681152/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202233.348709:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202233.348716:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202233.348720:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187680640:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202233.348741:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187680768:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202233.348762:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202233.348774:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202233/real 1788202233] req@ffff8e828793c380 x1875062187680768/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202233.348786:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187680768/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202233.348795:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202233.348798:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202233.348801:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187680768:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202233.348807:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187680896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202233.348825:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202233.348836:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202233/real 1788202233] req@ffff8e828793d880 x1875062187680896/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202233.348847:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187680896/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202233.348863:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202233.348869:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202233.348873:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187680896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202233.348882:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187681024:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202233.348901:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202233.348910:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202233/real 1788202233] req@ffff8e828793d500 x1875062187681024/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202233.348921:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187681024/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202233.348929:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202233.348932:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202233.348935:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187681024:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202233.348940:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187681152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202233.348953:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202233.348959:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202233/real 1788202233] req@ffff8e828793df80 x1875062187681152/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202233.348965:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793df80 x1875062187681152/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202233.348970:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202233.348972:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202233.348973:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187681152:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202234.052456:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202235.076482:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202236.100461:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202237.124482:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202238.148504:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202238.468482:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202238.468494:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202238.468500:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202238.468513:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3693 00000100:00080000:0.0:1788202238.468526:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202238.468536:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202238.468584:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187681280/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202238.468612:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202238.468616:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202238.468619:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202238.468624:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3693 00000100:00080000:0.0:1788202238.468628:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202238.468634:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202238.468638:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187681280:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788202238.468647:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202238.468664:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187681408/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202238.468678:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202238.468681:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202238.468684:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202238.468687:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3693 00000100:00080000:0.0:1788202238.468691:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202238.468695:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202238.468700:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202238.468712:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187681536/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202238.468721:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202238.468725:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202238.468727:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202238.468731:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3693 00000100:00080000:0.0:1788202238.468736:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202238.468741:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202238.468747:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202238.468753:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187681408:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202238.468762:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187681664/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202238.468781:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202238.468787:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788202238.468787:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187681536:192.168.201.121@tcp:38:kworker.0 00000100:00080000:0.0:1788202238.468792:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202238.468799:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3693 00000100:00080000:0.0:1788202238.468804:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202238.468809:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202238.468813:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187681664:192.168.201.121@tcp:8:kworker.0 00000008:00100000:0.0:1788202238.468816:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202238.468833:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187681792/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202238.468856:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187681792:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202239.172538:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202240.196455:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202241.220507:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788202241.540682:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:2.0:1788202241.540728:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788202241.540746:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788202241.540790:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788202241.540802:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788202241.540815:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788202241.540826:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:3.0:1788202241.540938:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202238/real 1788202241] req@ffff8e828793ca80 x1875062187681792/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202241.540964:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187681792/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202241.540978:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202241.540995:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202241.541000:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187681792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202241.541008:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202238/real 1788202241] req@ffff8e828793c700 x1875062187681664/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202241.541022:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187681664/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202241.541030:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202241.541033:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202241.541037:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187681664:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202241.541044:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202238/real 1788202241] req@ffff8e828793f480 x1875062187681408/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202241.541055:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187681408/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202241.541063:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202241.541067:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202241.541070:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187681408:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202241.541076:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202238/real 1788202241] req@ffff8e828793dc00 x1875062187681536/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202241.541085:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187681536/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202241.541749:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202241.541754:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202241.541758:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187681536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202241.541766:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202238/real 1788202241] req@ffff8e828793c000 x1875062187681280/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202241.541781:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187681280/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202241.541789:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202241.541792:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202241.541794:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187681280:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788202242.244466:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202243.268493:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202243.588429:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202243.588438:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202243.588443:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202243.588454:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3698 00000100:00080000:0.0:1788202243.588465:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202243.588471:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202243.588507:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187681920/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202243.588534:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202243.588538:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202243.588540:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202243.588544:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3698 00000100:00080000:0.0:1788202243.588549:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202243.588553:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202243.588563:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202243.588576:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187682048/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202243.588594:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187681920:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202243.588598:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202243.588610:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202243.588612:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202243.588617:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3698 00000100:00080000:0.0:1788202243.588620:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202243.588624:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202243.588630:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202243.588643:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187682176/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202243.588652:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202243.588656:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202243.588658:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202243.588663:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3698 00000100:00080000:0.0:1788202243.588668:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202243.588672:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202243.588678:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202243.588692:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187682304/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202243.588702:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202243.588714:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202243.588718:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202243.588721:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202243.588729:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3698 00000100:00080000:0.0:1788202243.588742:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202243.588747:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202243.588753:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202243.588765:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187682432/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202243.588805:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202243/real 1788202243] req@ffff8e828793f100 x1875062187681920/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202243.588832:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187681920/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202243.588847:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202243.588852:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202243.588856:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187681920:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202243.588888:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187682048:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202243.588915:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202243.588929:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202243/real 1788202243] req@ffff8e828793ce00 x1875062187682048/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202243.588942:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187682048/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202243.588965:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202243.588969:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202243.588973:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187682048:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202243.588980:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187682176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202243.589001:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202243.589012:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202243/real 1788202243] req@ffff8e828793f800 x1875062187682176/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202243.589023:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187682176/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202243.589032:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202243.589035:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202243.589038:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187682176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202243.589045:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187682304:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202243.589065:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202243.589076:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202243/real 1788202243] req@ffff8e828793e680 x1875062187682304/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202243.589087:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187682304/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202243.589094:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202243.589098:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202243.589101:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187682304:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202243.589107:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187682432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202243.589127:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202243.589137:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202243/real 1788202243] req@ffff8e828793e300 x1875062187682432/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202243.589148:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187682432/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202243.589156:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202243.589159:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202243.589162:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187682432:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202244.292436:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202245.316527:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202246.340447:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202247.364432:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202248.388432:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202248.708439:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202248.708447:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202248.708451:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202248.708461:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3703 00000100:00080000:1.0:1788202248.708471:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202248.708484:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202248.708514:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187682560/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202248.708537:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202248.708541:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202248.708543:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202248.708551:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3703 00000100:00100000:3.0:1788202248.708553:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187682560:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202248.708555:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202248.708562:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202248.708570:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202248.708582:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187682688/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202248.708592:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202248.708594:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202248.708596:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202248.708599:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3703 00000100:00080000:1.0:1788202248.708608:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202248.708611:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202248.708615:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788202248.708619:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202248.708627:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187682816/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202248.708637:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202248.708641:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202248.708643:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202248.708647:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3703 00000100:00080000:1.0:1788202248.708650:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202248.708654:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202248.708658:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202248/real 1788202248] req@ffff8e828d722d80 x1875062187682560/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788202248.708660:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202248.708676:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187682944/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202248.708683:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202248.708686:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202248.708687:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202248.708690:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3703 00000100:00080000:1.0:1788202248.708692:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202248.708696:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202248.708700:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202248.708708:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187683072/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202248.708738:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187682560/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202248.708746:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202248.708749:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202248.708751:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187682560:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202248.708769:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187682688:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202248.708906:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202248.708952:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202248/real 1788202248] req@ffff8e828d721180 x1875062187682688/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202248.708968:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187682688/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202248.708974:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202248.708977:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202248.708979:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187682688:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202248.708984:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187682816:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202248.709003:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202248.709012:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202248/real 1788202248] req@ffff8e828d720a80 x1875062187682816/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202248.709019:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187682816/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202248.709023:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202248.709025:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202248.709027:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187682816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202248.709029:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187682944:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202248.709039:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202248.709043:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202248/real 1788202248] req@ffff8e828d720380 x1875062187682944/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202248.709049:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187682944/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202248.709057:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202248.709059:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202248.709061:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187682944:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202248.709064:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187683072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202248.709072:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202248.709076:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202248/real 1788202248] req@ffff8e828d722300 x1875062187683072/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202248.709082:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187683072/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202248.709086:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202248.709088:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202248.709089:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187683072:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202249.412465:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202250.436454:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202251.460507:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202252.484534:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202253.508467:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202253.828431:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202253.828441:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202253.828446:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202253.828458:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3708 00000100:00080000:3.0:1788202253.828469:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202253.828476:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202253.828519:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187683200/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202253.828544:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202253.828548:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202253.828549:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202253.828552:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3708 00000100:00080000:3.0:1788202253.828554:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202253.828557:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202253.828567:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202253.828576:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187683328/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202253.828582:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187683200:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202253.828586:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202253.828589:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202253.828590:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202253.828592:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3708 00000100:00080000:3.0:1788202253.828594:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202253.828596:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202253.828598:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202253.828606:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187683456/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202253.828611:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202253.828613:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202253.828614:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202253.828616:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3708 00000100:00080000:3.0:1788202253.828618:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202253.828621:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202253.828625:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202253.828634:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187683584/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202253.828640:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202253.828642:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202253.828643:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202253.828645:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3708 00000100:00080000:3.0:1788202253.828647:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202253.828649:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202253.828651:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202253.828658:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187683712/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202253.828666:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202253.828718:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202253/real 1788202253] req@ffff8e8289539880 x1875062187683200/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202253.828739:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187683200/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202253.828748:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202253.828752:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202253.828755:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187683200:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202253.828801:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187683328:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202253.828820:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202253.828830:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202253/real 1788202253] req@ffff8e828953a680 x1875062187683328/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202253.828838:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187683328/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202253.828843:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202253.828846:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202253.828849:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187683328:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202253.828853:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187683456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202253.828864:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202253.828870:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202253/real 1788202253] req@ffff8e8289538a80 x1875062187683456/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202253.828877:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187683456/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202253.828882:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202253.828883:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202253.828885:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187683456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202253.828888:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187683584:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202253.828899:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202253.828904:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202253/real 1788202253] req@ffff8e828953b100 x1875062187683584/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202253.828912:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187683584/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202253.828916:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202253.828917:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202253.828919:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187683584:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788202253.828922:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187683712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202253.828931:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202253.828936:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202253/real 1788202253] req@ffff8e8289539f80 x1875062187683712/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202253.828943:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187683712/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202253.828948:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202253.828949:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202253.828951:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187683712:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202254.532549:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202255.556463:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202256.580490:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202257.604467:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202258.628483:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202258.948447:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202258.948454:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202258.948458:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202258.948468:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3714 00000100:00080000:2.0:1788202258.948477:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202258.948483:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202258.948515:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187683840/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202258.948539:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202258.948541:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202258.948543:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202258.948546:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3714 00000100:00080000:2.0:1788202258.948548:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202258.948551:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202258.948559:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202258.948567:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187683968/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202258.948574:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202258.948577:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202258.948580:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202258.948584:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3714 00000100:00080000:2.0:1788202258.948588:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788202258.948591:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187683840:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202258.948596:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202258.948604:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202258.948617:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187684096/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202258.948629:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202258.948633:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202258.948635:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202258.948639:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3714 00000100:00080000:2.0:1788202258.948643:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202258.948648:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:0.0:1788202258.948654:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:2.0:1788202258.948655:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202258.948671:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187684224/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202258.948694:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202258.948697:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202258.948699:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202258.948702:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3714 00000100:00080000:2.0:1788202258.948705:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202258.948709:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202258.948713:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202258.948715:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202258/real 1788202258] req@ffff8e82894e5880 x1875062187683840/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202258.948729:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187684352/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202258.948736:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187683840/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202258.948747:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202258.948750:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202258.948753:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187683840:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202258.948761:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187683968:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202258.948784:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202258.948797:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202258/real 1788202258] req@ffff8e82894e4000 x1875062187683968/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202258.948811:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187683968/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202258.948827:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202258.948831:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202258.948835:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187683968:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202258.948870:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187684096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202258.948890:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202258.948899:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202258/real 1788202258] req@ffff8e82894e6a00 x1875062187684096/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202258.948907:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187684096/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202258.948912:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202258.948915:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202258.948917:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187684096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202258.948921:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187684224:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202258.948932:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202258.948938:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202258/real 1788202258] req@ffff8e82894e6680 x1875062187684224/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202258.948945:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187684224/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202258.948950:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202258.948952:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202258.948954:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187684224:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788202258.948958:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187684352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202258.948968:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202258.948974:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202258/real 1788202258] req@ffff8e82894e7480 x1875062187684352/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202258.948981:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187684352/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202258.948986:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202258.948988:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202258.948990:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187684352:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202259.652436:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202260.676571:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202261.700487:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202262.724545:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202263.748473:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202264.068494:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202264.068507:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202264.068512:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202264.068523:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3719 00000100:00080000:3.0:1788202264.068536:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202264.068544:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202264.068614:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187684480/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202264.068646:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202264.068649:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202264.068650:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202264.068653:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3719 00000100:00080000:3.0:1788202264.068657:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202264.068661:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202264.068670:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202264.068683:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187684608/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202264.068691:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202264.068694:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202264.068695:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202264.068697:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3719 00000100:00080000:3.0:1788202264.068699:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202264.068702:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202264.068706:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202264.068712:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187684480:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202264.068719:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187684736/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202264.068728:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202264.068730:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202264.068731:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202264.068733:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3719 00000100:00080000:3.0:1788202264.068735:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202264.068737:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202264.068741:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202264.068749:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187684864/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202264.068761:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202264.068763:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202264.068765:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202264.068767:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3719 00000100:00080000:3.0:1788202264.068771:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202264.068773:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202264.068775:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202264.068782:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187684992/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202264.068800:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202264.068866:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202264/real 1788202264] req@ffff8e8289538380 x1875062187684480/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202264.068887:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187684480/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202264.068902:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202264.068907:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202264.068911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187684480:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202264.068920:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187684608:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202264.068944:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202264.068958:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202264/real 1788202264] req@ffff8e828953bb80 x1875062187684608/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202264.068970:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187684608/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202264.068979:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202264.068982:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202264.068986:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187684608:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202264.069013:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187684736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202264.069032:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202264.069044:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202264/real 1788202264] req@ffff8e8289539500 x1875062187684736/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202264.069057:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187684736/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202264.069065:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202264.069068:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202264.069071:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187684736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202264.069078:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187684864:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202264.069098:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202264.069109:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202264/real 1788202264] req@ffff8e8289539f80 x1875062187684864/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202264.069120:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187684864/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202264.069129:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202264.069132:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202264.069136:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187684864:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788202264.069143:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187684992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202264.069161:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202264.069172:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202264/real 1788202264] req@ffff8e828953b100 x1875062187684992/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202264.069183:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187684992/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202264.069192:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202264.069195:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202264.069198:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187684992:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202264.772510:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202265.796477:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202266.820494:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202267.844519:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202268.868495:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202269.188524:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202269.188554:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202269.188559:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202269.188570:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3724 00000100:00080000:2.0:1788202269.188591:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202269.188599:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202269.188649:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187685120/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202269.188677:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202269.188692:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202269.188693:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202269.188698:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3724 00000100:00080000:2.0:1788202269.188702:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202269.188714:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202269.188723:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202269.188726:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187685120:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202269.188738:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187685248/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202269.188756:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202269.188760:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202269.188762:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202269.188765:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3724 00000100:00080000:2.0:1788202269.188769:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202269.188773:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202269.188778:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202269.188790:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187685376/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202269.188799:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202269.188802:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202269.188805:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788202269.188810:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788202269.188814:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3724 00000100:00080000:2.0:1788202269.188819:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202269.188822:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202269.188827:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202269.188840:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187685504/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202269.188850:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202269.188853:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202269.188855:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202269.188859:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3724 00000100:00080000:2.0:1788202269.188863:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788202269.188868:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202269/real 1788202269] req@ffff8e82894e4e00 x1875062187685120/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202269.188871:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202269.188878:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202269.188894:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187685632/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202269.188906:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187685120/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202269.188919:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202269.188925:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202269.188930:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187685120:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202269.188969:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187685248:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202269.188990:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202269.189000:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202269/real 1788202269] req@ffff8e82894e7800 x1875062187685248/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202269.189008:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187685248/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202269.189014:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202269.189016:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202269.189019:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187685248:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202269.189022:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187685376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202269.189034:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202269.189040:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202269/real 1788202269] req@ffff8e82894e6300 x1875062187685376/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202269.189047:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187685376/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202269.189052:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202269.189054:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202269.189056:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187685376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202269.189060:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187685504:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202269.189070:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202269.189075:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202269/real 1788202269] req@ffff8e82894e4a80 x1875062187685504/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202269.189083:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187685504/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202269.189088:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202269.189090:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202269.189092:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187685504:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788202269.189095:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187685632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202269.189105:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202269.189110:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202269/real 1788202269] req@ffff8e82894e7100 x1875062187685632/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202269.189117:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187685632/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202269.189122:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202269.189124:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202269.189125:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187685632:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202269.892503:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202270.916492:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202271.940480:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202272.964457:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202273.988507:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202274.308375:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202274.308383:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202274.308387:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202274.308395:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3729 00000100:00080000:2.0:1788202274.308404:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202274.308423:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202274.308461:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187685760/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202274.308489:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202274.308494:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202274.308496:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202274.308500:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3729 00000100:00080000:2.0:1788202274.308504:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202274.308508:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202274.308516:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202274.308528:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187685888/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202274.308537:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187685760:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202274.308539:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202274.308547:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202274.308548:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202274.308551:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3729 00000100:00080000:2.0:1788202274.308554:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202274.308566:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202274.308570:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202274.308579:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187686016/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202274.308586:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202274.308589:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202274.308590:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202274.308595:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3729 00000800:00000100:0.0:1788202274.308597:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788202274.308599:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202274.308603:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202274.308607:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202274.308615:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187686144/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202274.308622:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202274.308624:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202274.308626:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202274.308629:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3729 00000100:00080000:2.0:1788202274.308632:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202274.308634:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202274.308637:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202274.308644:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202274/real 1788202274] req@ffff8e82894e6d80 x1875062187685760/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202274.308651:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187686272/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202274.308694:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187685760/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202274.308707:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202274.308711:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202274.308713:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187685760:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202274.308730:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187685888:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202274.308746:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202274.308754:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202274/real 1788202274] req@ffff8e82894e5f80 x1875062187685888/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202274.308762:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187685888/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202274.308768:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202274.308770:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202274.308772:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187685888:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202274.308775:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187686016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202274.308786:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202274.308792:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202274/real 1788202274] req@ffff8e82894e4380 x1875062187686016/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202274.308799:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187686016/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202274.308804:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202274.308806:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202274.308808:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187686016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202274.308812:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187686144:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202274.308822:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202274.308828:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202274/real 1788202274] req@ffff8e82894e5c00 x1875062187686144/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202274.308835:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187686144/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202274.308840:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202274.308842:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202274.308844:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187686144:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788202274.308847:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187686272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202274.308856:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202274.308861:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202274/real 1788202274] req@ffff8e82894e7100 x1875062187686272/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202274.308872:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187686272/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202274.308881:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202274.308884:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202274.308888:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187686272:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202275.012471:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202276.036499:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202277.060469:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202278.084522:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202279.108470:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202279.428479:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202279.428488:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202279.428491:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202279.428500:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3734 00000100:00080000:0.0:1788202279.428509:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202279.428516:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202279.428551:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187686400/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202279.428575:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202279.428578:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202279.428579:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202279.428583:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3734 00000100:00080000:0.0:1788202279.428586:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202279.428590:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202279.428598:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202279.428607:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187686528/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202279.428615:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202279.428618:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202279.428619:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202279.428622:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3734 00000100:00080000:0.0:1788202279.428624:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202279.428626:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202279.428637:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202279.428652:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187686656/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202279.428655:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187686400:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202279.428677:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202279.428680:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202279.428681:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202279.428684:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3734 00000100:00080000:0.0:1788202279.428687:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202279.428689:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202279.428692:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202279.428702:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187686784/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202279.428708:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202279.428710:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202279.428711:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202279.428714:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3734 00000100:00080000:0.0:1788202279.428716:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202279.428718:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202279.428721:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202279.428728:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187686912/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202279.428763:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202279.428827:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202279/real 1788202279] req@ffff8e828793e300 x1875062187686400/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202279.428849:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187686400/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202279.428863:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202279.428868:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202279.428873:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187686400:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202279.428882:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187686528:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202279.428907:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202279.428920:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202279/real 1788202279] req@ffff8e828793e680 x1875062187686528/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202279.428934:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187686528/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202279.428943:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202279.428946:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202279.428949:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187686528:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202279.428979:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187686656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202279.429000:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202279.429012:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202279/real 1788202279] req@ffff8e828793f800 x1875062187686656/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202279.429025:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187686656/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202279.429034:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202279.429038:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202279.429042:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187686656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202279.429048:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187686784:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202279.429066:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202279.429074:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202279/real 1788202279] req@ffff8e828793ce00 x1875062187686784/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202279.429084:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187686784/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202279.429100:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202279.429104:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202279.429107:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187686784:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202279.429115:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187686912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202279.429137:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202279.429147:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202279/real 1788202279] req@ffff8e828793f100 x1875062187686912/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202279.429155:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187686912/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202279.429161:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202279.429163:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202279.429165:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187686912:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202280.132569:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202281.156473:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202282.180489:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202283.204464:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202284.228499:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202284.548424:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202284.548432:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202284.548436:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202284.548445:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3739 00000100:00080000:0.0:1788202284.548454:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202284.548459:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202284.548490:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187687040/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202284.548513:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202284.548516:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202284.548521:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202284.548529:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3739 00000100:00080000:0.0:1788202284.548531:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202284.548534:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202284.548544:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202284.548553:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187687168/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202284.548564:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202284.548568:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202284.548570:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202284.548576:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3739 00000100:00100000:3.0:1788202284.548577:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187687040:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202284.548581:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202284.548587:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202284.548594:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202284.548607:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187687296/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202284.548616:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202284.548619:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202284.548620:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202284.548623:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3739 00000100:00080000:0.0:1788202284.548626:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202284.548629:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202284.548634:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788202284.548645:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202284.548649:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187687424/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202284.548662:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202284.548666:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202284.548668:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202284.548672:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3739 00000100:00080000:0.0:1788202284.548677:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202284.548682:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202284.548688:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202284.548689:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202284/real 1788202284] req@ffff8e828793c000 x1875062187687040/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202284.548703:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187687552/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202284.548713:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187687040/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202284.548726:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202284.548730:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202284.548733:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187687040:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202284.548740:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187687168:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202284.548761:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202284.548771:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202284/real 1788202284] req@ffff8e828793dc00 x1875062187687168/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202284.548781:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187687168/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202284.548788:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202284.548791:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202284.548795:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187687168:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202284.548820:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187687296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202284.548849:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202284.548863:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202284/real 1788202284] req@ffff8e828793f480 x1875062187687296/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202284.548878:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187687296/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202284.548888:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202284.548891:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202284.548896:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187687296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202284.548905:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187687424:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202284.548930:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202284.548947:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202284/real 1788202284] req@ffff8e828793c700 x1875062187687424/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202284.548961:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187687424/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202284.548969:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202284.548972:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202284.548976:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187687424:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202284.548983:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187687552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202284.549003:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202284.549075:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202284/real 1788202284] req@ffff8e828793ca80 x1875062187687552/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202284.549088:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187687552/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202284.549094:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202284.549097:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202284.549100:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187687552:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202285.252471:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202286.276548:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202287.300417:0:40714:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 2/2 locks to regular blp list, 1 blwis in pool 00010000:00010000:0.0:1788202287.300452:0:6172:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 2 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788202287.300464:0:6172:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 2/0,0 mode: PR/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc46a3 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788202287.300467:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202287.300570:0:6172:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26c5 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x7cf2114380cc4687 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788202287.300607:0:6172:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 1/0,0 mode: --/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc46a3 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788202287.300614:0:6172:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26c5 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc4687 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788202287.300618:0:6172:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788202287.300644:0:6172:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289463000/0x9ae3109cee9e26cc lrc: 0/0,0 mode: --/PR res: [0x2000090a1:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc46a3 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788202287.300658:0:6172:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8e8282a4b800 lock: ffff8e8289462200/0x9ae3109cee9e26c5 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x7cf2114380cc4687 expref: -99 pid: 42655 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788202287.300668:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202287.300671:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202287.300672:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202287.300680:0:2402:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8e828793df80 x1875062187687680/t0(0) o103->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1788202288.324456:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202289.348537:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202289.668420:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202289.668429:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202289.668432:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202289.668442:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3744 00000100:00080000:0.0:1788202289.668451:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202289.668459:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202289.668491:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187687808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202289.668518:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202289.668522:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202289.668524:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202289.668530:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3744 00000100:00080000:0.0:1788202289.668536:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202289.668542:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202289.668552:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202289.668552:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187687808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202289.668569:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187687936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202289.668577:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202289.668580:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202289.668582:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202289.668585:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3744 00000100:00080000:0.0:1788202289.668588:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202289.668590:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202289.668594:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202289.668602:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187688064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202289.668608:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202289.668611:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202289.668612:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202289.668617:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3744 00000800:00000100:3.0:1788202289.668618:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202289.668621:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202289.668634:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202289.668638:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202289.668648:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187688192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202289.668659:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202289.668664:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788202289.668664:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202289/real 1788202289] req@ffff8e828793d500 x1875062187687808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202289.668668:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202289.668676:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3744 00000100:00080000:0.0:1788202289.668682:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202289.668686:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202289.668689:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202289.668690:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187687808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202289.668703:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187688320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202289.668709:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202289.668716:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202289.668721:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187687808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202289.668740:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187687936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202289.668756:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202289.668773:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202289/real 1788202289] req@ffff8e828793d880 x1875062187687936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202289.668782:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187687936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202289.668788:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202289.668790:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202289.668792:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187687936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202289.668797:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187688064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202289.668809:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202289.668815:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202289/real 1788202289] req@ffff8e828793c380 x1875062187688064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202289.668822:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187688064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202289.668828:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202289.668829:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202289.668831:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187688064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202289.668835:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187688192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202289.668846:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202289.668851:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202289/real 1788202289] req@ffff8e828793fb80 x1875062187688192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202289.668859:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187688192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202289.668864:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202289.668865:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202289.668867:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187688192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202289.668870:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187688320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202289.668879:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202289.668885:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202289/real 1788202289] req@ffff8e828793ca80 x1875062187688320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202289.668892:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187688320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202289.668896:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202289.668898:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202289.668900:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187688320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202290.372484:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202291.396484:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202292.420455:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202293.444486:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202294.468460:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202294.788463:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202294.788474:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202294.788479:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202294.788491:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3749 00000100:00080000:0.0:1788202294.788502:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202294.788511:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202294.788549:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187688448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202294.788586:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202294.788591:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202294.788593:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202294.788602:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3749 00000100:00080000:0.0:1788202294.788607:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202294.788612:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202294.788622:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202294.788640:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187688576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202294.788640:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187688448:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202294.788658:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202294.788662:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202294.788663:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202294.788667:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3749 00000100:00080000:0.0:1788202294.788671:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202294.788676:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202294.788682:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202294.788696:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187688704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202294.788707:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202294.788712:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1788202294.788713:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202294.788715:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202294.788722:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3749 00000100:00080000:0.0:1788202294.788727:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202294.788732:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202294.788738:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202294.788751:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187688832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202294.788755:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202294/real 1788202294] req@ffff8e828793c700 x1875062187688448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202294.788788:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202294.788792:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202294.788794:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788202294.788796:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187688448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202294.788799:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3749 00000100:00080000:0.0:1788202294.788806:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202294.788814:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:3.0:1788202294.788815:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788202294.788820:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788202294.788822:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202294.788829:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187688448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202294.788836:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187688960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202294.788859:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187688576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202294.788895:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202294.788907:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202294/real 1788202294] req@ffff8e828793f480 x1875062187688576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202294.788916:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187688576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202294.788922:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202294.788924:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202294.788931:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187688576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202294.788936:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187688704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202294.788950:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202294.788957:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202294/real 1788202294] req@ffff8e828793dc00 x1875062187688704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202294.788968:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187688704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202294.788983:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202294.788987:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202294.788990:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187688704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202294.788997:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187688832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202294.789019:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202294.789028:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202294/real 1788202294] req@ffff8e828793c000 x1875062187688832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202294.789036:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187688832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202294.789042:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202294.789044:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202294.789068:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187688832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202294.789074:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187688960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202294.789089:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202294.789097:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202294/real 1788202294] req@ffff8e828793f100 x1875062187688960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202294.789105:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187688960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202294.789110:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202294.789112:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202294.789115:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187688960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202295.492490:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202296.516463:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202297.540485:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202298.564476:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202299.588492:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202299.908432:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202299.908440:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202299.908444:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202299.908452:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3754 00000100:00080000:2.0:1788202299.908462:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202299.908467:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202299.908500:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187689088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202299.908523:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202299.908525:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202299.908527:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202299.908530:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3754 00000100:00080000:2.0:1788202299.908532:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202299.908535:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202299.908542:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202299.908551:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187689216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202299.908559:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202299.908562:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202299.908564:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788202299.908568:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187689088:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202299.908571:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3754 00000100:00080000:2.0:1788202299.908577:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202299.908582:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202299.908596:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202299.908612:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187689344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202299.908623:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202299.908627:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202299.908629:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788202299.908633:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788202299.908637:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3754 00000100:00080000:2.0:1788202299.908643:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202299.908649:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202299.908655:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202299.908669:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187689472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202299.908679:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202299.908683:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202299.908685:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788202299.908689:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202299/real 1788202299] req@ffff8e82894e4a80 x1875062187689088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202299.908693:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3754 00000100:00080000:2.0:1788202299.908701:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202299.908708:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202299.908715:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202299.908719:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187689088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202299.908731:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1788202299.908734:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187689600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202299.908739:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202299.908748:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187689088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202299.908776:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187689216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202299.908796:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202299.908806:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202299/real 1788202299] req@ffff8e82894e6300 x1875062187689216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202299.908815:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187689216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202299.908821:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202299.908823:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202299.908825:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187689216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202299.908829:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187689344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202299.908840:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202299.908847:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202299/real 1788202299] req@ffff8e82894e7800 x1875062187689344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202299.908854:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187689344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202299.908859:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202299.908861:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202299.908863:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187689344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202299.908867:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187689472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202299.908889:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202299.908895:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202299/real 1788202299] req@ffff8e82894e4e00 x1875062187689472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202299.908903:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187689472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202299.908908:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202299.908910:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202299.908912:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187689472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202299.908915:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187689600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202299.908925:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202299.908931:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202299/real 1788202299] req@ffff8e82894e7480 x1875062187689600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202299.908938:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187689600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202299.908943:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202299.908945:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202299.908947:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187689600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202300.612543:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202301.636485:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202302.660457:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202303.684482:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202304.708442:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202305.028430:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202305.028438:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202305.028441:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202305.028450:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3760 00000100:00080000:1.0:1788202305.028458:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202305.028465:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202305.028496:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187689728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202305.028518:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202305.028521:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202305.028522:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202305.028526:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3760 00000100:00080000:1.0:1788202305.028528:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202305.028531:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202305.028538:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202305.028547:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187689856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202305.028554:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202305.028557:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202305.028558:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202305.028561:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3760 00000100:00080000:1.0:1788202305.028565:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202305.028568:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202305.028571:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202305.028577:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187689984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202305.028584:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202305.028586:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202305.028587:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202305.028601:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3760 00000100:00100000:3.0:1788202305.028604:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187689728:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202305.028605:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202305.028618:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202305.028627:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202305.028635:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187690112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202305.028641:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202305.028643:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202305.028645:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202305.028647:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3760 00000100:00080000:1.0:1788202305.028649:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202305.028651:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202305.028655:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202305.028663:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187690240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202305.028754:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187689856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202305.028805:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187689984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202305.028821:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187690112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202305.028840:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187690240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202305.732494:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202306.756491:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202307.780507:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788202308.100580:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:2.0:1788202308.100623:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788202308.100639:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788202308.100681:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788202308.100693:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788202308.100701:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788202308.100712:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:3.0:1788202308.100742:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202305/real 1788202308] req@ffff8e828d720a80 x1875062187689984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202308.100770:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187689984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202308.100783:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202308.100788:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202308.100793:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187689984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202308.100802:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202305/real 1788202308] req@ffff8e828d721180 x1875062187690112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202308.100816:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187690112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202308.100863:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202308.100868:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202308.100872:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187690112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202308.100880:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202305/real 1788202308] req@ffff8e828d722d80 x1875062187690240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202308.100894:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187690240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202308.100903:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202308.100906:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202308.100911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187690240:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202308.100918:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202305/real 1788202308] req@ffff8e828d722300 x1875062187689728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202308.100932:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187689728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202308.100941:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202308.100945:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202308.100949:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187689728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202308.100956:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202305/real 1788202308] req@ffff8e828d720380 x1875062187689856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202330 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202308.100969:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187689856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202330 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202308.100978:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202308.100982:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202308.100986:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187689856:192.168.201.121@tcp:38:kworker.0 00010000:00010000:0.0:1788202308.804461:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202309.828494:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202310.148426:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202310.148437:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202310.148441:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202310.148452:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3765 00000100:00080000:1.0:1788202310.148463:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202310.148471:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202310.148509:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187690368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202310.148546:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202310.148550:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202310.148552:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202310.148558:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3765 00000100:00080000:1.0:1788202310.148562:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202310.148568:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202310.148579:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202310.148594:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187690496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202310.148607:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202310.148613:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788202310.148614:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187690368:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202310.148615:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202310.148625:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3765 00000100:00080000:1.0:1788202310.148630:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202310.148633:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202310.148639:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202310.148653:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187690624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202310.148663:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202310.148667:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202310.148668:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202310.148673:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3765 00000100:00080000:1.0:1788202310.148677:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202310.148682:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202310.148688:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788202310.148689:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202310.148704:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187690752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202310.148716:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202310.148721:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202310.148723:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202310.148727:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3765 00000100:00080000:1.0:1788202310.148731:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788202310.148737:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202310/real 1788202310] req@ffff8e828d720e00 x1875062187690368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202310.148739:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202310.148748:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202310.148766:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187690368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202310.148786:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187690880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202310.148808:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202310.148815:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202310.148819:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187690368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202310.148848:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187690496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202310.148877:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202310.148891:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202310/real 1788202310] req@ffff8e828d720000 x1875062187690496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202310.148904:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187690496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202310.148911:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202310.148914:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202310.148917:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187690496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202310.148923:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187690624:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202310.148941:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202310.148951:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202310/real 1788202310] req@ffff8e828d721500 x1875062187690624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202310.148962:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187690624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202310.148969:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202310.148971:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202310.148975:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187690624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202310.148981:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187690752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202310.148998:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202310.149010:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202310/real 1788202310] req@ffff8e828d723800 x1875062187690752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202310.149022:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187690752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202310.149043:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202310.149046:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202310.149050:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187690752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202310.149056:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187690880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202310.149076:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202310.149088:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202310/real 1788202310] req@ffff8e828d722680 x1875062187690880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202310.149102:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187690880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202335 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202310.149110:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202310.149114:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202310.149117:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187690880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202310.852453:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202311.876496:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202312.900426:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202313.924460:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202314.948445:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202315.268431:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202315.268442:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202315.268446:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202315.268456:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3770 00000100:00080000:0.0:1788202315.268467:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202315.268475:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202315.268515:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187691008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202315.268544:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202315.268548:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202315.268550:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202315.268555:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3770 00000100:00080000:0.0:1788202315.268559:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202315.268563:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202315.268573:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202315.268585:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187691136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202315.268595:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202315.268599:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202315.268601:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202315.268606:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3770 00000100:00080000:0.0:1788202315.268610:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202315.268614:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202315.268620:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202315.268624:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187691008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202315.268634:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187691264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202315.268647:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202315.268651:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202315.268654:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202315.268659:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3770 00000100:00080000:0.0:1788202315.268664:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202315.268668:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202315.268674:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788202315.268684:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202315.268690:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187691392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202315.268717:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202315.268721:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202315.268723:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202315.268729:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3770 00000100:00080000:0.0:1788202315.268733:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202315.268738:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202315.268744:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202315.268744:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202315/real 1788202315] req@ffff8e828793ce00 x1875062187691008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202315.268763:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187691520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202315.268766:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187691008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202315.268794:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202315.268799:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202315.268803:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187691008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202315.268809:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187691136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202315.268827:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202315.268836:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202315/real 1788202315] req@ffff8e828793f800 x1875062187691136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202315.268844:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187691136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202315.268854:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202315.268856:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202315.268858:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187691136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202315.268876:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187691264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202315.268886:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202315.268893:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202315/real 1788202315] req@ffff8e828793e680 x1875062187691264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202315.268900:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187691264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202315.268905:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202315.268906:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202315.268908:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187691264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202315.268912:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187691392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202315.268922:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202315.268927:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202315/real 1788202315] req@ffff8e828793e300 x1875062187691392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202315.268934:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187691392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202315.268939:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202315.268941:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202315.268943:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187691392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202315.268947:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187691520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202315.268957:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202315.268964:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202315/real 1788202315] req@ffff8e828793f100 x1875062187691520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202315.268975:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187691520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202340 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202315.268983:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202315.268986:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202315.268989:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187691520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202315.972445:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202316.996490:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202318.020470:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202319.044504:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202320.068440:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202320.388431:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202320.388443:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202320.388447:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202320.388455:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3775 00000100:00080000:2.0:1788202320.388466:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202320.388471:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202320.388535:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187691648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202320.388561:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202320.388565:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202320.388567:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202320.388571:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3775 00000100:00080000:2.0:1788202320.388575:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202320.388579:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202320.388588:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202320.388594:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187691648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202320.388602:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187691776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202320.388628:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202320.388631:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202320.388633:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202320.388636:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3775 00000100:00080000:2.0:1788202320.388639:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202320.388642:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202320.388647:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202320.388660:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187691904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202320.388671:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788202320.388672:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202320.388678:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202320.388681:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202320.388686:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3775 00000100:00080000:2.0:1788202320.388691:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202320.388696:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202320.388701:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202320.388709:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202320/real 1788202320] req@ffff8e82894e6680 x1875062187691648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202320.388717:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187692032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202320.388727:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187691648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202320.388731:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202320.388740:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202320.388742:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202320.388744:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202320.388748:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3775 00000100:00080000:1.0:1788202320.388752:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:2.0:1788202320.388753:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202320.388757:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187691648:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202320.388759:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202320.388767:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202320.388777:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187692160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202320.388785:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187691776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202320.388816:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202320.388831:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202320/real 1788202320] req@ffff8e82894e6a00 x1875062187691776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202320.388846:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187691776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202320.388855:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202320.388859:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202320.388863:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187691776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202320.388871:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187691904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202320.388896:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202320.388913:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202320/real 1788202320] req@ffff8e82894e4000 x1875062187691904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202320.388928:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187691904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202320.388939:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202320.388944:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202320.388949:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187691904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202320.388959:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187692032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202320.388984:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202320.388997:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202320/real 1788202320] req@ffff8e82894e5880 x1875062187692032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202320.389005:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187692032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202320.389011:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202320.389013:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202320.389015:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187692032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202320.389026:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187692160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202320.389040:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202320.389046:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202320/real 1788202320] req@ffff8e82894e7480 x1875062187692160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202320.389053:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187692160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202320.389059:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202320.389061:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202320.389063:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187692160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202321.092471:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202322.116476:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202323.140488:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202324.164525:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202325.188485:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202325.508492:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202325.508503:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202325.508508:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202325.508520:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3780 00000100:00080000:3.0:1788202325.508554:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202325.508563:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202325.508619:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187692288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202325.508644:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202325.508647:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202325.508649:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202325.508652:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3780 00000100:00080000:3.0:1788202325.508655:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202325.508671:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202325.508682:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202325.508705:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187692416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202325.508718:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788202325.508723:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187692288:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202325.508724:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202325.508727:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202325.508735:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3780 00000100:00080000:3.0:1788202325.508739:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202325.508742:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202325.508747:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202325.508758:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187692544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202325.508766:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202325.508769:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202325.508770:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202325.508773:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3780 00000100:00080000:3.0:1788202325.508776:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202325.508778:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202325.508782:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202325.508804:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187692672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202325.508814:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202325.508819:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202325.508821:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202325.508825:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3780 00000800:00000100:1.0:1788202325.508831:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788202325.508832:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202325.508841:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202325.508846:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202325.508860:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187692800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202325.508895:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202325/real 1788202325] req@ffff8e8289538a80 x1875062187692288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202325.508929:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187692288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202325.508942:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202325.508947:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202325.508952:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187692288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202325.508992:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187692416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202325.509017:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202325.509032:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202325/real 1788202325] req@ffff8e828953a680 x1875062187692416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202325.509046:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187692416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202325.509056:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202325.509059:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202325.509063:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187692416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202325.509070:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187692544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202325.509090:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202325.509104:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202325/real 1788202325] req@ffff8e8289539880 x1875062187692544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202325.509116:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187692544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202325.509125:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202325.509129:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202325.509132:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187692544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202325.509139:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187692672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202325.509161:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202325.509182:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202325/real 1788202325] req@ffff8e828953b100 x1875062187692672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202325.509195:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187692672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202325.509204:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202325.509208:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202325.509212:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187692672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202325.509229:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187692800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202325.509251:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202325.509263:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202325/real 1788202325] req@ffff8e8289539f80 x1875062187692800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202325.509274:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187692800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202325.509283:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202325.509287:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202325.509291:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187692800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202326.212441:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202327.236476:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202328.260521:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202329.284495:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202330.308499:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202330.628456:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202330.628473:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202330.628478:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202330.628488:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3785 00000100:00080000:2.0:1788202330.628500:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202330.628508:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202330.628543:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187692928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202330.628570:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202330.628573:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202330.628574:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202330.628577:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3785 00000100:00080000:2.0:1788202330.628580:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202330.628583:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202330.628590:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202330.628598:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187693056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202330.628605:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202330.628608:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202330.628610:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202330.628613:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3785 00000100:00080000:2.0:1788202330.628616:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202330.628618:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202330.628621:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202330.628628:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187693184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202330.628632:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187692928:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202330.628638:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202330.628641:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202330.628642:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202330.628645:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3785 00000100:00080000:2.0:1788202330.628647:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202330.628650:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202330.628653:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202330.628661:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187693312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202330.628667:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202330.628670:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202330.628672:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202330.628674:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3785 00000100:00080000:2.0:1788202330.628677:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202330.628679:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202330.628682:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202330.628689:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187693440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202330.628696:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202330.628740:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202330/real 1788202330] req@ffff8e82894e4e00 x1875062187692928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202330.628758:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187692928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202330.628766:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202330.628769:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202330.628772:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187692928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202330.628777:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187693056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202330.628793:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202330.628801:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202330/real 1788202330] req@ffff8e82894e7800 x1875062187693056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202330.628809:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187693056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202330.628815:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202330.628817:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202330.628819:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187693056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202330.628837:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187693184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202330.628855:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202330.628866:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202330/real 1788202330] req@ffff8e82894e6300 x1875062187693184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202330.628878:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187693184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202330.628893:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202330.628897:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202330.628900:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187693184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202330.628907:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187693312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202330.628928:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202330.628939:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202330/real 1788202330] req@ffff8e82894e4a80 x1875062187693312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202330.628947:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187693312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202330.628953:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202330.628955:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202330.628957:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187693312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202330.628961:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187693440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202330.628974:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202330.628980:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202330/real 1788202330] req@ffff8e82894e7100 x1875062187693440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202330.628987:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187693440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202330.628992:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202330.628994:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202330.628996:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187693440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202331.332445:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202332.356528:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202333.380474:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202334.404498:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202335.428472:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202335.748510:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202335.748519:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202335.748523:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202335.748533:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3790 00000100:00080000:1.0:1788202335.748545:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202335.748554:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202335.748593:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187693568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202335.748621:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202335.748625:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202335.748627:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202335.748632:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3790 00000100:00080000:1.0:1788202335.748637:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202335.748641:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202335.748651:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202335.748666:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187693696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202335.748674:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187693568:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202335.748677:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202335.748682:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202335.748684:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202335.748687:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3790 00000100:00080000:1.0:1788202335.748690:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202335.748693:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202335.748697:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202335.748707:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187693824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202335.748714:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202335.748716:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202335.748717:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202335.748720:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3790 00000100:00080000:1.0:1788202335.748725:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202335.748727:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202335.748730:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788202335.748734:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202335.748740:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187693952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202335.748749:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202335.748752:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202335.748753:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202335.748756:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3790 00000100:00080000:1.0:1788202335.748758:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202335.748761:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202335.748764:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202335.748772:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187694080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202335.748776:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202335/real 1788202335] req@ffff8e828d722680 x1875062187693568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202335.748820:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187693568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202335.748830:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202335.748833:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202335.748836:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187693568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202335.748856:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187693696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202335.748870:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202335.748878:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202335/real 1788202335] req@ffff8e828d723800 x1875062187693696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202335.748886:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187693696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202335.748893:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202335.748895:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202335.748897:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187693696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202335.748900:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187693824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202335.748911:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202335.748917:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202335/real 1788202335] req@ffff8e828d721500 x1875062187693824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202335.748924:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187693824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202335.748929:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202335.748931:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202335.748932:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187693824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202335.748936:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187693952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202335.748946:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202335.748952:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202335/real 1788202335] req@ffff8e828d720000 x1875062187693952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202335.748958:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187693952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202335.748963:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202335.748965:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202335.748967:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187693952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202335.748971:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187694080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202335.748981:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202335.748986:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202335/real 1788202335] req@ffff8e828d720e00 x1875062187694080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202335.748993:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187694080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202335.748998:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202335.749000:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202335.749001:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187694080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202336.452497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202337.476469:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202338.500558:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202339.524489:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202340.548477:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202340.868428:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202340.868438:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202340.868443:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202340.868453:0:37:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3795 00000100:00080000:1.0:1788202340.868465:0:37:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202340.868474:0:37:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202340.868510:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187694208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202340.868534:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202340.868538:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202340.868540:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202340.868545:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3795 00000100:00080000:1.0:1788202340.868549:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202340.868554:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202340.868564:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202340.868566:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187694208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202340.868582:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187694336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202340.868595:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202340.868598:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202340.868601:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202340.868605:0:37:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3795 00000100:00080000:1.0:1788202340.868609:0:37:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202340.868615:0:37:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202340.868621:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788202340.868629:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202340.868636:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187694464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202340.868648:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202340.868652:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202340.868654:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202340.868658:0:37:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3795 00000100:00080000:1.0:1788202340.868663:0:37:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788202340.868664:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202340/real 1788202340] req@ffff8e828d720380 x1875062187694208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202340.868673:0:37:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202340.868681:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202340.868685:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187694208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202340.868695:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1788202340.868696:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187694592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202340.868701:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202340.868708:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187694208:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202340.868719:0:37:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202340.868723:0:37:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202340.868725:0:37:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202340.868729:0:37:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3795 00000100:00100000:3.0:1788202340.868733:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187694336:192.168.201.121@tcp:38:kworker.0 00000100:00080000:1.0:1788202340.868735:0:37:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202340.868743:0:37:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202340.868749:0:37:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788202340.868754:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202340.868765:0:37:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187694720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202340.868767:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202340/real 1788202340] req@ffff8e828d722300 x1875062187694336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202340.868798:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187694336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202340.868805:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202340.868807:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202340.868810:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187694336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202340.868816:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187694464:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202340.868830:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202340.868841:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202340/real 1788202340] req@ffff8e828d722d80 x1875062187694464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202340.868854:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187694464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202340.868863:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202340.868866:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202340.868870:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187694464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202340.868876:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187694592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202340.868898:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202340.868910:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202340/real 1788202340] req@ffff8e828d721180 x1875062187694592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202340.868918:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187694592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202340.868924:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202340.868927:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202340.868929:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187694592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202340.868941:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187694720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202340.868954:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202340.868961:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202340/real 1788202340] req@ffff8e828d720a80 x1875062187694720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202340.868968:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187694720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202340.868973:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202340.868975:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202340.868976:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187694720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202341.572461:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202342.596504:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202343.620455:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202344.644498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202345.668543:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202345.988465:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202345.988475:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202345.988480:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202345.988491:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3801 00000100:00080000:0.0:1788202345.988502:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202345.988511:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202345.988548:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187694848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202345.988580:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202345.988584:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202345.988586:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202345.988591:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3801 00000100:00080000:0.0:1788202345.988595:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202345.988600:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202345.988610:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202345.988626:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187694976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202345.988645:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202345.988650:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202345.988653:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202345.988658:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3801 00000100:00100000:3.0:1788202345.988667:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187694848:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202345.988668:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202345.988678:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202345.988689:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202345.988706:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187695104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202345.988716:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202345.988720:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202345.988722:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202345.988727:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3801 00000100:00080000:0.0:1788202345.988731:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202345.988736:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202345.988742:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202345.988756:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187695232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202345.988768:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202345.988772:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202345.988777:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788202345.988779:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202345.988786:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3801 00000100:00080000:0.0:1788202345.988793:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202345.988798:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202345.988805:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202345.988821:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187695360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202345.988869:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202345/real 1788202345] req@ffff8e828793c000 x1875062187694848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202345.988895:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187694848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202345.988908:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202345.988913:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202345.988918:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187694848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202345.988951:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187694976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202345.988978:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202345.988992:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202345/real 1788202345] req@ffff8e828793dc00 x1875062187694976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202345.989006:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187694976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202345.989016:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202345.989020:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202345.989024:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187694976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202345.989031:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187695104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202345.989051:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202345.989062:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202345/real 1788202345] req@ffff8e828793f480 x1875062187695104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202345.989076:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187695104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202345.989085:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202345.989089:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202345.989093:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187695104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202345.989099:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187695232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202345.989120:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202345.989131:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202345/real 1788202345] req@ffff8e828793c700 x1875062187695232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202345.989145:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187695232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202345.989154:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202345.989158:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202345.989162:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187695232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202345.989184:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187695360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202345.989206:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202345.989218:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202345/real 1788202345] req@ffff8e828793ca80 x1875062187695360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202345.989232:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187695360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202345.989241:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202345.989245:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202345.989248:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187695360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202346.692555:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202347.716480:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202348.740476:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202349.764510:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202350.788529:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202351.108426:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202351.108437:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202351.108442:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202351.108452:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3806 00000100:00080000:0.0:1788202351.108464:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202351.108472:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202351.108511:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187695488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202351.108541:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202351.108546:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202351.108548:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202351.108553:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3806 00000100:00080000:0.0:1788202351.108558:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202351.108563:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202351.108574:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202351.108584:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187695488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202351.108591:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187695616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202351.108612:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202351.108617:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202351.108618:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202351.108623:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3806 00000100:00080000:0.0:1788202351.108628:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202351.108633:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202351.108638:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202351.108652:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187695744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202351.108656:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202351.108668:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202351.108672:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202351.108675:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202351.108680:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3806 00000100:00080000:0.0:1788202351.108684:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202351.108689:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202351.108695:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202351.108707:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187695872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202351.108712:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202351/real 1788202351] req@ffff8e828793fb80 x1875062187695488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202351.108730:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202351.108736:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202351.108738:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788202351.108744:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187695488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202351.108745:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3806 00000100:00080000:0.0:1788202351.108753:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202351.108760:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202351.108769:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788202351.108776:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202351.108782:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202351.108786:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187696000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202351.108791:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187695488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202351.108823:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187695616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202351.108847:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202351.108860:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202351/real 1788202351] req@ffff8e828793c380 x1875062187695616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202351.108874:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187695616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202351.108883:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202351.108887:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202351.108891:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187695616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202351.108898:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187695744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202351.108916:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202351.108927:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202351/real 1788202351] req@ffff8e828793d880 x1875062187695744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202351.108940:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187695744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202351.108949:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202351.108952:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202351.108956:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187695744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202351.108962:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187695872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202351.108981:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202351.108992:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202351/real 1788202351] req@ffff8e828793d500 x1875062187695872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202351.109004:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187695872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202351.109013:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202351.109017:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202351.109021:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187695872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202351.109027:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187696000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202351.109045:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202351.109056:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202351/real 1788202351] req@ffff8e828793ca80 x1875062187696000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202376 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202351.109068:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187696000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202376 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202351.109107:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202351.109111:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202351.109115:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187696000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202351.812508:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202352.836460:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202353.860532:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202354.884463:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202355.908512:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202356.228406:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202356.228415:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202356.228419:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202356.228429:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3811 00000100:00080000:0.0:1788202356.228439:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202356.228445:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202356.228477:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187696128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202356.228505:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202356.228508:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202356.228509:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202356.228513:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3811 00000100:00080000:0.0:1788202356.228515:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202356.228518:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202356.228526:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202356.228539:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187696128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202356.228540:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187696256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202356.228562:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202356.228566:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202356.228568:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202356.228572:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3811 00000100:00080000:0.0:1788202356.228576:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202356.228581:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202356.228593:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202356.228611:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187696384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202356.228620:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202356.228623:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1788202356.228627:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202356.228628:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202356.228636:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3811 00000100:00080000:0.0:1788202356.228640:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202356.228644:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202356.228650:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202356.228659:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187696512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202356.228667:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202356.228669:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202356.228671:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202356.228674:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3811 00000100:00080000:0.0:1788202356.228677:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202356.228680:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202356.228684:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202356.228693:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187696640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202356.228756:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202356/real 1788202356] req@ffff8e828793c700 x1875062187696128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202356.228781:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187696128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202356.228794:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202356.228799:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202356.228804:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187696128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202356.228831:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187696256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202356.228856:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202356.228868:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202356/real 1788202356] req@ffff8e828793f480 x1875062187696256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202356.228877:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187696256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202356.228883:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202356.228886:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202356.228888:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187696256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202356.228893:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187696384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202356.228908:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202356.228921:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202356/real 1788202356] req@ffff8e828793dc00 x1875062187696384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202356.228936:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187696384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202356.228945:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202356.228949:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202356.228953:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187696384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202356.228962:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187696512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202356.228986:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202356.229018:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202356/real 1788202356] req@ffff8e828793c000 x1875062187696512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202356.229035:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187696512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202356.229044:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202356.229047:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202356.229049:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187696512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202356.229054:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187696640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202356.229072:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202356.229080:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202356/real 1788202356] req@ffff8e828793f100 x1875062187696640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202381 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202356.229093:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187696640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202381 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202356.229098:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202356.229100:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202356.229102:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187696640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202356.932464:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202357.956468:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202358.980481:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202360.004499:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202361.028450:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202361.348422:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202361.348429:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202361.348433:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202361.348442:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3816 00000100:00080000:2.0:1788202361.348449:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202361.348456:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202361.348488:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187696768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202361.348510:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202361.348513:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202361.348514:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202361.348517:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3816 00000100:00080000:2.0:1788202361.348519:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202361.348522:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202361.348528:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202361.348537:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187696896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202361.348549:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788202361.348552:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187696768:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202361.348553:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202361.348557:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202361.348563:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3816 00000100:00080000:2.0:1788202361.348565:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202361.348567:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202361.348570:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202361.348578:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187697024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202361.348585:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202361.348588:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202361.348590:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202361.348594:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3816 00000100:00080000:2.0:1788202361.348608:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202361.348613:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202361.348619:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788202361.348620:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202361.348638:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187697152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202361.348649:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202361.348652:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202361.348654:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202361.348659:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3816 00000100:00100000:3.0:1788202361.348662:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202361/real 1788202361] req@ffff8e82894e5c00 x1875062187696768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202361.348665:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202361.348676:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202361.348682:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187696768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788202361.348683:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788202361.348696:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202361.348699:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202361.348704:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187697280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202361.348705:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187696768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202361.348732:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187696896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202361.348754:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202361.348768:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202361/real 1788202361] req@ffff8e82894e4380 x1875062187696896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202361.348783:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187696896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202361.348791:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202361.348793:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202361.348795:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187696896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202361.348800:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187697024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202361.348815:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202361.348831:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202361/real 1788202361] req@ffff8e82894e5f80 x1875062187697024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202361.348844:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187697024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202361.348859:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202361.348862:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202361.348866:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187697024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202361.348872:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187697152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202361.348893:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202361.348927:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202361/real 1788202361] req@ffff8e82894e6d80 x1875062187697152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202361.348936:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187697152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202361.348941:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202361.348944:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202361.348945:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187697152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202361.348949:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187697280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202361.348963:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202361.348970:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202361/real 1788202361] req@ffff8e82894e7100 x1875062187697280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202361.348977:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187697280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202361.348982:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202361.348984:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202361.348986:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187697280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202362.052547:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202363.076443:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202364.100492:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202365.124440:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202366.148556:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202366.468515:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202366.468529:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202366.468535:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202366.468548:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3821 00000100:00080000:1.0:1788202366.468562:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202366.468571:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202366.468625:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187697408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202366.468660:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202366.468665:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202366.468667:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202366.468672:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3821 00000100:00080000:1.0:1788202366.468676:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202366.468681:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202366.468694:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202366.468716:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187697536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202366.468729:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202366.468734:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788202366.468735:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187697408:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202366.468737:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202366.468747:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3821 00000100:00080000:1.0:1788202366.468752:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202366.468756:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202366.468762:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202366.468776:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187697664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202366.468783:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202366.468785:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202366.468787:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202366.468790:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3821 00000100:00080000:1.0:1788202366.468793:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202366.468796:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202366.468800:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202366.468808:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187697792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202366.468830:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202366.468832:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202366.468835:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202366.468838:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3821 00000100:00080000:1.0:1788202366.468841:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202366.468844:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202366.468848:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788202366.468851:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202366.468861:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187697920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202366.468911:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202366/real 1788202366] req@ffff8e828d720a80 x1875062187697408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202366.468934:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187697408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202366.468947:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202366.468952:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202366.468957:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187697408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202366.468989:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187697536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202366.469015:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202366.469029:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202366/real 1788202366] req@ffff8e828d721180 x1875062187697536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202366.469044:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187697536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202366.469053:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202366.469057:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202366.469061:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187697536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202366.469068:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187697664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202366.469089:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202366.469102:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202366/real 1788202366] req@ffff8e828d722d80 x1875062187697664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202366.469114:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187697664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202366.469123:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202366.469126:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202366.469130:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187697664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202366.469136:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187697792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202366.469157:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202366.469168:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202366/real 1788202366] req@ffff8e828d722300 x1875062187697792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202366.469181:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187697792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202366.469189:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202366.469193:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202366.469196:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187697792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202366.469201:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187697920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202366.469219:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202366.469229:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202366/real 1788202366] req@ffff8e828d720380 x1875062187697920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202366.469241:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187697920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202366.469250:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202366.469253:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202366.469256:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187697920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202367.172489:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202368.196502:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202369.220573:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202370.244525:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202371.268584:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202371.588428:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202371.588444:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202371.588451:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202371.588464:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3826 00000100:00080000:3.0:1788202371.588479:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202371.588489:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202371.588545:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187698048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202371.588578:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202371.588581:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202371.588583:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202371.588587:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3826 00000100:00080000:3.0:1788202371.588590:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202371.588593:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202371.588603:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202371.588614:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187698176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202371.588622:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202371.588625:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202371.588629:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202371.588630:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187698048:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202371.588634:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3826 00000100:00080000:3.0:1788202371.588640:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202371.588643:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202371.588646:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202371.588654:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187698304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202371.588661:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202371.588665:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202371.588668:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202371.588672:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3826 00000100:00080000:3.0:1788202371.588676:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202371.588680:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202371.588687:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202371.588713:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187698432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202371.588735:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202371.588738:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202371.588740:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202371.588743:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3826 00000100:00080000:3.0:1788202371.588746:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202371.588750:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202371.588755:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202371.588770:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187698560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202371.588789:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187698176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202371.588831:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187698304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202371.588862:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187698432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202371.588898:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187698560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202372.292533:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202373.316560:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202374.340472:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788202374.660706:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788202374.660757:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788202374.660776:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202374.660824:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202374.660834:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202374.660843:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202374.660853:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:2.0:1788202374.660889:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202371/real 1788202374] req@ffff8e828953bb80 x1875062187698176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202374.660930:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187698176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202374.660942:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202374.660947:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202374.660951:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187698176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202374.660960:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202371/real 1788202374] req@ffff8e8289538380 x1875062187698304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202374.660977:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187698304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202374.660988:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202374.660991:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202374.660995:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187698304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202374.661001:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202371/real 1788202374] req@ffff8e8289539f80 x1875062187698432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202374.661015:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187698432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202374.661030:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202374.661035:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202374.661039:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187698432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202374.661046:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202371/real 1788202374] req@ffff8e828953b100 x1875062187698560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202374.661059:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187698560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202374.661067:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202374.661071:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202374.661074:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187698560:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202374.661081:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202371/real 1788202374] req@ffff8e8289539500 x1875062187698048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202374.661093:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187698048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202374.661102:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202374.661106:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202374.661109:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187698048:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788202375.364496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202376.388468:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202376.708450:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202376.708459:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202376.708462:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202376.708471:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3831 00000100:00080000:1.0:1788202376.708480:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202376.708487:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202376.708518:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187698688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202376.708556:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202376.708560:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202376.708562:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202376.708567:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3831 00000100:00080000:1.0:1788202376.708570:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202376.708576:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202376.708586:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202376.708602:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187698688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202376.708603:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187698816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202376.708621:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202376.708625:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202376.708627:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202376.708632:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3831 00000100:00080000:1.0:1788202376.708664:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202376.708670:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202376.708675:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788202376.708679:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202376.708693:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187698944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202376.708705:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202376.708708:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202376.708711:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202376.708715:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3831 00000100:00080000:1.0:1788202376.708719:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202376.708722:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202376.708727:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202376.708732:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202376/real 1788202376] req@ffff8e828d720e00 x1875062187698688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202376.708741:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187699072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202376.708756:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202376.708762:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788202376.708762:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187698688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202376.708765:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202376.708775:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3831 00000100:00080000:2.0:1788202376.708783:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202376.708784:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202376.708793:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:1.0:1788202376.708794:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202376.708803:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202376.708803:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187698688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202376.708820:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187699200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202376.708838:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187698816:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202376.708883:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202376.708898:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202376/real 1788202376] req@ffff8e828d720000 x1875062187698816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202376.708911:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187698816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202376.708918:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202376.708921:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202376.708924:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187698816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202376.708931:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187698944:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202376.708952:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202376.708963:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202376/real 1788202376] req@ffff8e828d721500 x1875062187698944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202376.708975:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187698944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202376.708982:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202376.708985:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202376.708988:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187698944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202376.708995:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187699072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202376.709015:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202376.709027:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202376/real 1788202376] req@ffff8e828d723800 x1875062187699072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202376.709039:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187699072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202376.709047:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202376.709050:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202376.709053:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187699072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202376.709059:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187699200:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202376.709079:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202376.709092:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202376/real 1788202376] req@ffff8e828d722680 x1875062187699200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202376.709105:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187699200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202376.709114:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202376.709117:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202376.709120:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187699200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202377.412430:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202378.436478:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202379.460447:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202380.484436:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202381.508464:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202381.828370:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202381.828377:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202381.828381:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202381.828390:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3836 00000100:00080000:1.0:1788202381.828399:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202381.828406:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202381.828435:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187699328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202381.828455:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202381.828458:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202381.828459:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202381.828462:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3836 00000100:00080000:1.0:1788202381.828464:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202381.828467:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202381.828476:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202381.828485:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187699456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202381.828492:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202381.828495:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788202381.828495:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187699328:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202381.828497:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202381.828504:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3836 00000100:00080000:1.0:1788202381.828505:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202381.828507:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202381.828510:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202381.828517:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187699584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202381.828524:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202381.828526:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202381.828527:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202381.828529:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3836 00000100:00080000:1.0:1788202381.828530:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202381.828533:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202381.828536:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202381.828557:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187699712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202381.828564:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202381.828567:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202381.828568:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202381.828571:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3836 00000100:00080000:1.0:1788202381.828574:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202381.828577:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202381.828580:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202381.828589:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187699840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202381.828598:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202381.828648:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202381/real 1788202381] req@ffff8e828d722680 x1875062187699328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202381.828669:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187699328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202381.828682:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202381.828686:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202381.828690:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187699328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202381.828713:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187699456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202381.828736:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202381.828750:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202381/real 1788202381] req@ffff8e828d723800 x1875062187699456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202381.828765:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187699456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202381.828779:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202381.828781:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202381.828784:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187699456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202381.828788:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187699584:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202381.828804:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202381.828813:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202381/real 1788202381] req@ffff8e828d721500 x1875062187699584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202381.828820:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187699584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202381.828837:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202381.828839:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202381.828841:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187699584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202381.828845:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187699712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202381.828860:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202381.828871:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202381/real 1788202381] req@ffff8e828d720000 x1875062187699712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202381.828884:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187699712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202381.828892:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202381.828896:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202381.828900:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187699712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202381.828906:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187699840:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202381.828925:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202381.828934:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202381/real 1788202381] req@ffff8e828d720e00 x1875062187699840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202381.828941:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187699840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202381.828947:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202381.828949:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202381.828951:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187699840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202382.532448:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202383.556488:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202384.580513:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202385.604502:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202386.628516:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202386.948449:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202386.948471:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202386.948476:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202386.948487:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3842 00000100:00080000:3.0:1788202386.948500:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202386.948508:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202386.948550:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187699968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202386.948578:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202386.948582:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202386.948583:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202386.948587:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3842 00000100:00080000:3.0:1788202386.948591:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202386.948594:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202386.948603:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202386.948617:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187700096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202386.948631:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187699968:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202386.948633:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202386.948645:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202386.948652:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202386.948657:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3842 00000100:00080000:3.0:1788202386.948659:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202386.948661:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202386.948665:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202386.948675:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187700224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202386.948682:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202386.948685:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202386.948687:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202386.948692:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3842 00000100:00080000:3.0:1788202386.948704:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202386.948708:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202386.948713:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202386.948726:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187700352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202386.948733:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788202386.948745:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202386.948749:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202386.948750:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202386.948756:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3842 00000100:00080000:3.0:1788202386.948759:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202386.948763:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202386.948766:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202386.948772:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187700480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202386.948818:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202386/real 1788202386] req@ffff8e8289539880 x1875062187699968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202386.948842:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187699968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202386.948851:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202386.948855:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202386.948859:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187699968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202386.948884:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187700096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202386.948911:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202386.948923:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202386/real 1788202386] req@ffff8e828953a680 x1875062187700096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202386.948931:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187700096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202386.948937:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202386.948940:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202386.948942:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187700096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202386.948946:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187700224:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202386.948960:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202386.948966:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202386/real 1788202386] req@ffff8e8289538a80 x1875062187700224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202386.948974:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187700224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202386.948978:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202386.948980:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202386.948982:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187700224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202386.948986:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187700352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202386.948996:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202386.949002:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202386/real 1788202386] req@ffff8e8289539500 x1875062187700352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202386.949009:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187700352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202386.949015:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202386.949017:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202386.949019:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187700352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202386.949031:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187700480:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202386.949043:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202386.949049:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202386/real 1788202386] req@ffff8e828953b100 x1875062187700480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202386.949056:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187700480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202386.949061:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202386.949063:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202386.949065:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187700480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202387.652498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202388.676538:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202389.700514:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202390.724515:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202391.748489:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202392.068479:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202392.068488:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202392.068493:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202392.068504:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3847 00000100:00080000:1.0:1788202392.068515:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202392.068523:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202392.068560:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187700608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202392.068591:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202392.068595:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202392.068597:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202392.068604:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3847 00000100:00080000:1.0:1788202392.068608:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202392.068613:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202392.068623:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202392.068635:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187700608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202392.068642:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187700736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202392.068655:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202392.068658:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202392.068661:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202392.068665:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3847 00000100:00080000:1.0:1788202392.068669:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202392.068674:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202392.068679:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202392.068691:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187700864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202392.068698:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202392.068700:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202392.068703:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788202392.068705:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788202392.068709:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3847 00000100:00080000:1.0:1788202392.068714:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202392.068717:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202392.068721:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202392.068729:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187700992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202392.068738:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202392.068742:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788202392.068745:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202392/real 1788202392] req@ffff8e828d720380 x1875062187700608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202392.068746:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202392.068756:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3847 00000100:00080000:1.0:1788202392.068763:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788202392.068767:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187700608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202392.068769:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202392.068778:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788202392.068781:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202392.068785:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202392.068789:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187700608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202392.068794:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187701120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202392.068823:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187700736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202392.068852:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202392.068870:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202392/real 1788202392] req@ffff8e828d722300 x1875062187700736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202392.068889:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187700736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202392.068899:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202392.068903:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202392.068906:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187700736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202392.068912:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187700864:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202392.068931:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202392.068944:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202392/real 1788202392] req@ffff8e828d722d80 x1875062187700864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202392.068953:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187700864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202392.068958:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202392.068962:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202392.068964:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187700864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202392.068967:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187700992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202392.068981:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202392.068989:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202392/real 1788202392] req@ffff8e828d721180 x1875062187700992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202392.068998:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187700992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202392.069005:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202392.069008:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202392.069009:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187700992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788202392.069015:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187701120:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202392.069027:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202392.069038:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202392/real 1788202392] req@ffff8e828d720a80 x1875062187701120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202417 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202392.069050:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187701120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202417 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202392.069055:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202392.069058:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202392.069062:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187701120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202392.772498:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202393.796493:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202394.820550:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202395.844479:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202396.868471:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202397.188403:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202397.188414:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202397.188418:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202397.188428:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3852 00000100:00080000:2.0:1788202397.188438:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202397.188446:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202397.188481:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187701248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202397.188509:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202397.188514:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202397.188516:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202397.188520:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3852 00000100:00080000:2.0:1788202397.188524:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202397.188527:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202397.188537:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202397.188549:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187701376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202397.188557:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202397.188561:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788202397.188564:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187701248:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202397.188564:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202397.188570:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3852 00000100:00080000:2.0:1788202397.188576:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202397.188578:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202397.188581:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202397.188589:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187701504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202397.188603:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202397.188605:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202397.188610:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202397.188612:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3852 00000100:00080000:2.0:1788202397.188615:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202397.188618:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202397.188622:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202397.188629:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187701632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202397.188638:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202397.188641:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202397.188643:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788202397.188649:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788202397.188650:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3852 00000100:00080000:2.0:1788202397.188658:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202397.188663:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202397.188669:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202397.188683:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187701760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202397.188698:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202397/real 1788202397] req@ffff8e82894e4a80 x1875062187701248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202397.188717:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187701248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202397.188727:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202397.188731:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202397.188735:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187701248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202397.188756:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187701376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202397.188790:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202397.188803:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202397/real 1788202397] req@ffff8e82894e6300 x1875062187701376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202397.188816:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187701376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202397.188825:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202397.188827:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202397.188829:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187701376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202397.188834:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187701504:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202397.188854:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202397.188877:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202397/real 1788202397] req@ffff8e82894e7800 x1875062187701504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202397.188889:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187701504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202397.188897:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202397.188900:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202397.188904:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187701504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202397.188911:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187701632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202397.188931:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202397.188943:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202397/real 1788202397] req@ffff8e82894e4e00 x1875062187701632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202397.188956:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187701632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202397.188965:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202397.188968:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202397.188971:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187701632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788202397.188977:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187701760:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202397.188992:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202397.189001:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202397/real 1788202397] req@ffff8e82894e7480 x1875062187701760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202397.189012:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187701760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202422 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202397.189020:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202397.189022:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202397.189024:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187701760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202397.892470:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202398.916553:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202399.940478:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202400.964529:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202401.988467:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202402.308460:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202402.308475:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202402.308479:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202402.308490:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3857 00000100:00080000:3.0:1788202402.308512:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202402.308522:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202402.308572:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187701888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202402.308600:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202402.308604:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202402.308605:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202402.308609:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3857 00000100:00080000:3.0:1788202402.308612:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202402.308616:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202402.308632:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202402.308647:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187701888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202402.308649:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187702016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202402.308663:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202402.308667:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202402.308669:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202402.308671:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3857 00000100:00080000:3.0:1788202402.308675:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202402.308679:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202402.308682:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202402.308697:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187702144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202402.308704:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202402.308707:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202402.308709:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202402.308712:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3857 00000100:00080000:3.0:1788202402.308723:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202402.308726:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202402.308729:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202402.308737:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187702272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202402.308744:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788202402.308750:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788202402.308750:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202402.308754:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202402.308758:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3857 00000100:00080000:3.0:1788202402.308761:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202402.308764:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202402.308767:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202402.308774:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187702400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202402.308832:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202402/real 1788202402] req@ffff8e8289539f80 x1875062187701888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202402.308856:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187701888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202402.308870:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202402.308875:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202402.308879:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187701888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202402.308920:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187702016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202402.308948:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202402.308972:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202402/real 1788202402] req@ffff8e8289538380 x1875062187702016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202402.308987:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187702016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202402.308997:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202402.309000:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202402.309004:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187702016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202402.309011:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187702144:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202402.309034:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202402.309047:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202402/real 1788202402] req@ffff8e828953bb80 x1875062187702144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202402.309060:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187702144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202402.309069:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202402.309072:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202402.309076:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187702144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202402.309091:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187702272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202402.309115:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202402.309127:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202402/real 1788202402] req@ffff8e828953b100 x1875062187702272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202402.309147:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187702272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202402.309155:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202402.309159:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202402.309162:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187702272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202402.309169:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187702400:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202402.309223:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202402.309238:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202402/real 1788202402] req@ffff8e8289539500 x1875062187702400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202402.309251:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187702400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202402.309260:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202402.309264:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202402.309267:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187702400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202403.012545:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202404.036439:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202405.060533:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202406.084531:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202407.108488:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202407.428443:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202407.428454:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202407.428458:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202407.428467:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3862 00000100:00080000:1.0:1788202407.428477:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202407.428482:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202407.428516:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187702528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202407.428539:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202407.428542:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202407.428544:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202407.428547:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3862 00000100:00080000:1.0:1788202407.428549:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202407.428551:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202407.428559:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202407.428566:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187702656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202407.428573:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202407.428576:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202407.428577:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202407.428579:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3862 00000100:00080000:1.0:1788202407.428581:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202407.428583:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202407.428585:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202407.428592:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187702784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202407.428601:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202407.428605:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202407.428607:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202407.428610:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187702528:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202407.428615:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3862 00000100:00080000:1.0:1788202407.428621:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202407.428626:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202407.428631:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202407.428645:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187702912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202407.428656:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202407.428658:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202407.428660:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202407.428663:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3862 00000100:00080000:1.0:1788202407.428666:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202407.428670:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202407.428675:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788202407.428681:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202407.428690:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187703040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202407.428722:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202407/real 1788202407] req@ffff8e828d720a80 x1875062187702528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202407.428736:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187702528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202407.428749:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202407.428752:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202407.428755:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187702528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202407.428759:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187702656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202407.428773:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202407.428781:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202407/real 1788202407] req@ffff8e828d721180 x1875062187702656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202407.428789:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187702656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202407.428794:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202407.428796:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202407.428798:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187702656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202407.428817:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187702784:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202407.428827:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202407.428833:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202407/real 1788202407] req@ffff8e828d722d80 x1875062187702784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202407.428840:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187702784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202407.428845:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202407.428847:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202407.428849:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187702784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202407.428853:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187702912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202407.428864:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202407.428874:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202407/real 1788202407] req@ffff8e828d722300 x1875062187702912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202407.428887:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187702912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202407.428895:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202407.428899:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202407.428902:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187702912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202407.428909:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187703040:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202407.428928:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202407.428939:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202407/real 1788202407] req@ffff8e828d720380 x1875062187703040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202407.428949:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187703040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202407.428956:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202407.428960:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202407.428963:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187703040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202408.132487:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202409.156484:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202410.180492:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202411.204497:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202412.228510:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202412.548402:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202412.548412:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202412.548416:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202412.548425:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3867 00000100:00080000:1.0:1788202412.548434:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202412.548440:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202412.548472:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187703168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202412.548495:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202412.548498:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202412.548499:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202412.548502:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3867 00000100:00080000:1.0:1788202412.548504:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202412.548507:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202412.548514:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202412.548521:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187703296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202412.548528:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202412.548531:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202412.548532:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202412.548535:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3867 00000100:00080000:1.0:1788202412.548537:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202412.548539:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202412.548544:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202412.548544:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187703168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202412.548552:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187703424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202412.548559:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202412.548562:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202412.548563:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202412.548566:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3867 00000100:00080000:1.0:1788202412.548568:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202412.548571:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202412.548574:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202412.548589:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187703552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202412.548595:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202412.548597:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202412.548598:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788202412.548604:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788202412.548605:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3867 00000100:00080000:1.0:1788202412.548610:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202412.548613:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202412.548616:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202412.548623:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187703680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202412.548654:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202412/real 1788202412] req@ffff8e828d720e00 x1875062187703168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202412.548676:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187703168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202412.548687:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202412.548692:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202412.548696:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187703168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202412.548704:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187703296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202412.548724:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202412.548734:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202412/real 1788202412] req@ffff8e828d720000 x1875062187703296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202412.548743:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187703296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202412.548760:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202412.548763:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202412.548765:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187703296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202412.548785:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187703424:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202412.548800:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202412.548807:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202412/real 1788202412] req@ffff8e828d721500 x1875062187703424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202412.548815:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187703424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202412.548825:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202412.548827:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202412.548829:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187703424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202412.548833:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187703552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202412.548852:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202412.548859:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202412/real 1788202412] req@ffff8e828d723800 x1875062187703552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202412.548866:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187703552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202412.548872:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202412.548874:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202412.548876:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187703552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202412.548879:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187703680:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202412.548890:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202412.548896:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202412/real 1788202412] req@ffff8e828d722680 x1875062187703680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202412.548903:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187703680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202412.548907:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202412.548909:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202412.548911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187703680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202413.252483:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202414.276510:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202415.300479:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202416.324568:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202417.348466:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202417.668409:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202417.668417:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202417.668420:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202417.668428:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3872 00000100:00080000:1.0:1788202417.668438:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202417.668443:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202417.668477:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187703808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202417.668499:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202417.668502:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202417.668503:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202417.668506:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3872 00000100:00080000:1.0:1788202417.668508:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202417.668511:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202417.668518:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202417.668527:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187703936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202417.668534:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202417.668537:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202417.668540:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202417.668545:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3872 00000100:00080000:1.0:1788202417.668548:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202417.668550:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202417.668553:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202417.668558:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187704064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202417.668566:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202417.668568:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202417.668569:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202417.668573:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3872 00000100:00080000:1.0:1788202417.668577:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202417.668584:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202417.668584:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187703808:192.168.201.121@tcp:250:kworker.0 00000008:00100000:1.0:1788202417.668596:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202417.668605:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187704192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202417.668610:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202417.668612:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202417.668613:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202417.668616:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3872 00000100:00080000:1.0:1788202417.668619:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202417.668622:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202417.668631:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202417.668644:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187704320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202417.668681:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202417.668736:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202417/real 1788202417] req@ffff8e828d722680 x1875062187703808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202417.668753:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187703808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202417.668763:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202417.668767:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202417.668770:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187703808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202417.668777:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187703936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202417.668793:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202417.668801:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202417/real 1788202417] req@ffff8e828d723800 x1875062187703936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202417.668809:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187703936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202417.668816:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202417.668819:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202417.668821:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187703936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202417.668845:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187704064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202417.668857:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202417.668864:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202417/real 1788202417] req@ffff8e828d721500 x1875062187704064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202417.668872:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187704064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202417.668878:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202417.668880:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202417.668882:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187704064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202417.668886:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187704192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202417.668897:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202417.668903:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202417/real 1788202417] req@ffff8e828d720000 x1875062187704192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202417.668911:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187704192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202417.668916:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202417.668918:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202417.668920:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187704192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202417.668949:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187704320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202417.668964:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202417.668972:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202417/real 1788202417] req@ffff8e828d720e00 x1875062187704320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202417.668979:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187704320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202417.668985:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202417.668987:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202417.668990:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187704320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202418.372583:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202419.396441:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202420.420487:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202421.444531:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202422.468503:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202422.788440:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202422.788450:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202422.788455:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202422.788466:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3877 00000100:00080000:1.0:1788202422.788479:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202422.788486:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202422.788532:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187704448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202422.788560:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202422.788563:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202422.788565:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202422.788568:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3877 00000100:00080000:1.0:1788202422.788571:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202422.788573:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202422.788583:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202422.788595:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187704576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202422.788602:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202422.788605:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202422.788608:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202422.788610:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3877 00000100:00080000:1.0:1788202422.788612:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202422.788615:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202422.788618:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202422.788629:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187704704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202422.788633:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187704448:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202422.788638:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202422.788644:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202422.788645:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202422.788651:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3877 00000100:00080000:1.0:1788202422.788654:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202422.788656:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202422.788660:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202422.788673:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187704832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202422.788679:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202422.788681:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202422.788683:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202422.788687:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3877 00000100:00080000:1.0:1788202422.788690:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202422.788694:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202422.788696:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202422.788705:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187704960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202422.788730:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202422.788773:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202422/real 1788202422] req@ffff8e828d720380 x1875062187704448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202422.788787:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187704448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202422.788796:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202422.788799:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202422.788802:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187704448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202422.788808:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187704576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202422.788824:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202422.788847:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202422/real 1788202422] req@ffff8e828d722300 x1875062187704576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202422.788857:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187704576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202422.788866:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202422.788870:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202422.788873:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187704576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202422.788901:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187704704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202422.788922:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202422.788931:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202422/real 1788202422] req@ffff8e828d722d80 x1875062187704704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202422.788939:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187704704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202422.788944:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202422.788946:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202422.788948:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187704704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202422.788953:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187704832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202422.788964:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202422.788970:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202422/real 1788202422] req@ffff8e828d721180 x1875062187704832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202422.788977:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187704832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202422.788982:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202422.788984:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202422.788986:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187704832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202422.788989:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187704960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202422.788999:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202422.789005:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202422/real 1788202422] req@ffff8e828d720a80 x1875062187704960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202422.789012:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187704960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202422.789017:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202422.789019:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202422.789021:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187704960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202423.492541:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202424.516524:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202425.540463:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202426.564522:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202427.588498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202427.908452:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202427.908461:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202427.908466:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202427.908477:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3882 00000100:00080000:3.0:1788202427.908490:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202427.908498:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202427.908544:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187705088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202427.908573:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202427.908576:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202427.908577:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202427.908581:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3882 00000100:00080000:3.0:1788202427.908584:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202427.908588:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202427.908597:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202427.908610:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187705216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202427.908619:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202427.908622:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788202427.908623:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187705088:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202427.908624:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202427.908629:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3882 00000100:00080000:3.0:1788202427.908635:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202427.908640:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202427.908644:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202427.908661:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187705344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202427.908672:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202427.908676:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202427.908678:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202427.908685:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3882 00000100:00080000:3.0:1788202427.908690:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202427.908696:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202427.908702:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788202427.908709:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202427.908760:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202427/real 1788202427] req@ffff8e8289538a80 x1875062187705088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202427.908776:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187705088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202427.908789:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202427.908793:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202427.908802:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187705088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202427.908802:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187705472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202427.908828:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202427.908833:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202427.908835:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202427.908841:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3882 00000100:00080000:3.0:1788202427.908848:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202427.908850:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187705216:192.168.201.121@tcp:38:kworker.0 00000100:00080000:3.0:1788202427.908856:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202427.908864:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202427.908879:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187705600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202427.908890:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202427.908906:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202427/real 1788202427] req@ffff8e828953a680 x1875062187705216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202427.908921:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187705216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202427.908933:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202427.908935:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202427.908937:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187705216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202427.908942:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187705344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202427.908962:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202427.908977:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202427/real 1788202427] req@ffff8e8289539880 x1875062187705344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202427.908990:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187705344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202427.908996:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202427.908999:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202427.909002:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187705344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202427.909011:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187705472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202427.909041:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202427.909064:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202427/real 1788202427] req@ffff8e8289539500 x1875062187705472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202427.909077:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187705472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202427.909086:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202427.909090:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202427.909093:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187705472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202427.909138:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187705600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202427.909159:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202427.909166:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202427/real 1788202427] req@ffff8e828953b100 x1875062187705600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202427.909176:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187705600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202427.909191:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202427.909194:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202427.909196:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187705600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202428.612500:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202429.636642:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202430.660465:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202431.684493:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202432.708517:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202433.028427:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202433.028439:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202433.028443:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202433.028454:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3888 00000100:00080000:3.0:1788202433.028466:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202433.028474:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202433.028515:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187705728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202433.028546:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202433.028550:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202433.028552:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202433.028557:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3888 00000100:00080000:3.0:1788202433.028561:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202433.028565:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202433.028574:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202433.028586:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187705856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202433.028594:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202433.028597:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202433.028598:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202433.028603:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3888 00000100:00100000:1.0:1788202433.028604:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187705728:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202433.028605:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202433.028610:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202433.028617:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202433.028622:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187705984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202433.028628:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202433.028630:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202433.028632:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202433.028634:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3888 00000100:00080000:3.0:1788202433.028636:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202433.028638:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202433.028641:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202433.028648:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187706112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202433.028653:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202433.028665:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202433.028666:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202433.028668:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3888 00000100:00080000:3.0:1788202433.028671:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202433.028674:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202433.028678:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202433.028685:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187706240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202433.028709:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202433.028756:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202433/real 1788202433] req@ffff8e828953bb80 x1875062187705728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202433.028771:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187705728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202433.028779:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202433.028782:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202433.028786:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187705728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202433.028807:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187705856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202433.028824:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202433.028832:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202433/real 1788202433] req@ffff8e8289538380 x1875062187705856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202433.028839:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187705856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202433.028844:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202433.028846:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202433.028848:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187705856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202433.028851:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187705984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202433.028860:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202433.028873:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202433/real 1788202433] req@ffff8e8289539f80 x1875062187705984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202433.028887:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187705984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202433.028892:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202433.028894:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202433.028896:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187705984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202433.028899:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187706112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202433.028910:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202433.028915:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202433/real 1788202433] req@ffff8e828953b100 x1875062187706112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202433.028922:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187706112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202433.028926:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202433.028928:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202433.028929:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187706112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202433.028932:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187706240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202433.028940:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202433.028945:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202433/real 1788202433] req@ffff8e8289539500 x1875062187706240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202458 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202433.028951:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187706240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202458 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202433.028960:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202433.028962:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202433.028964:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187706240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202433.732512:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202434.756534:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202435.780515:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202436.804448:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202437.828433:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202438.148420:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202438.148429:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202438.148432:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202438.148441:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3893 00000100:00080000:0.0:1788202438.148451:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202438.148457:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202438.148495:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187706368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202438.148514:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202438.148517:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202438.148518:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202438.148521:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3893 00000100:00080000:0.0:1788202438.148524:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202438.148527:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202438.148537:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202438.148559:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187706496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202438.148580:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187706368:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202438.148601:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202438.148605:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202438.148608:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202438.148613:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3893 00000100:00080000:0.0:1788202438.148618:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202438.148622:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202438.148629:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202438.148643:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187706624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202438.148652:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202438.148655:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202438.148656:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202438.148659:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3893 00000100:00080000:0.0:1788202438.148662:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202438.148664:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202438.148669:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202438.148678:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187706752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202438.148685:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202438.148688:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202438.148690:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202438.148692:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3893 00000100:00080000:0.0:1788202438.148695:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202438.148698:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202438.148702:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202438.148707:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187706496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202438.148714:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187706880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202438.148747:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187706624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202438.148772:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187706752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202438.148800:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187706880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202438.852438:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202439.876460:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202440.900506:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788202441.220678:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:2.0:1788202441.220734:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788202441.220758:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788202441.220824:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788202441.220836:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788202441.220847:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788202441.220858:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788202441.220894:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202438/real 1788202441] req@ffff8e828793f100 x1875062187706880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202441.220939:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187706880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202441.220954:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202441.220961:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202441.220966:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187706880:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202441.220980:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202438/real 1788202441] req@ffff8e828793e680 x1875062187706496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202441.220995:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187706496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202441.221010:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202441.221015:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202441.221019:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187706496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202441.221032:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202438/real 1788202441] req@ffff8e828793f800 x1875062187706624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202441.221047:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187706624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202441.221056:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202441.221060:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202441.221063:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187706624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202441.221071:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202438/real 1788202441] req@ffff8e828793ce00 x1875062187706752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202441.221085:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187706752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202441.221094:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202441.221098:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202441.221101:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187706752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202441.221110:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202438/real 1788202441] req@ffff8e828793e300 x1875062187706368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202463 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202441.221124:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187706368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202463 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202441.221133:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202441.221136:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202441.221140:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187706368:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788202441.924528:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202442.948542:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202443.268503:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202443.268518:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202443.268525:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202443.268540:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3898 00000100:00080000:3.0:1788202443.268554:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202443.268564:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202443.268607:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187707008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202443.268644:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202443.268649:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202443.268651:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202443.268656:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3898 00000100:00080000:3.0:1788202443.268660:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202443.268665:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202443.268677:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202443.268695:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187707136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202443.268705:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187707008:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202443.268725:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202443.268729:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202443.268731:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202443.268737:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3898 00000100:00080000:3.0:1788202443.268741:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202443.268746:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202443.268752:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202443.268764:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187707264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202443.268782:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202443.268785:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202443.268787:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202443.268792:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3898 00000100:00080000:3.0:1788202443.268797:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202443.268802:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:1.0:1788202443.268808:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:3.0:1788202443.268810:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202443.268828:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187707392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202443.268838:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202443.268842:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202443.268843:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202443.268847:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3898 00000100:00080000:3.0:1788202443.268851:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202443.268856:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202443.268863:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202443.268874:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202443/real 1788202443] req@ffff8e8289539880 x1875062187707008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202443.268879:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187707520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202443.268934:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187707008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202443.268951:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202443.268957:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202443.268962:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187707008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202443.269000:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187707136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202443.269028:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202443.269044:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202443/real 1788202443] req@ffff8e828953a680 x1875062187707136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202443.269064:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187707136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202443.269073:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202443.269076:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202443.269080:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187707136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202443.269087:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187707264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202443.269109:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202443.269122:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202443/real 1788202443] req@ffff8e8289538a80 x1875062187707264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202443.269136:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187707264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202443.269145:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202443.269149:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202443.269153:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187707264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202443.269160:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187707392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202443.269185:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202443.269198:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202443/real 1788202443] req@ffff8e8289539500 x1875062187707392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202443.269219:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187707392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202443.269234:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202443.269238:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202443.269241:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187707392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202443.269248:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187707520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202443.269280:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202443.269291:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202443/real 1788202443] req@ffff8e828953b100 x1875062187707520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202468 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202443.269302:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187707520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202468 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202443.269349:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202443.269353:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202443.269356:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187707520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202443.972417:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202444.996438:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202446.020455:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202447.044494:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202448.068459:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202448.388456:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202448.388468:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202448.388472:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202448.388485:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3903 00000100:00080000:3.0:1788202448.388497:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202448.388505:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202448.388551:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187707648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202448.388581:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202448.388584:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202448.388586:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202448.388589:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3903 00000100:00080000:3.0:1788202448.388591:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202448.388594:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202448.388604:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202448.388615:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187707776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202448.388623:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202448.388626:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202448.388627:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202448.388631:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3903 00000100:00080000:3.0:1788202448.388634:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202448.388638:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202448.388644:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202448.388646:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187707648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202448.388668:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187707904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202448.388681:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202448.388686:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202448.388688:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202448.388693:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3903 00000100:00080000:3.0:1788202448.388697:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202448.388702:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202448.388708:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202448.388721:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187708032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202448.388733:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202448.388737:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1788202448.388742:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788202448.388742:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202448.388751:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3903 00000100:00080000:3.0:1788202448.388756:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202448.388761:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202448.388767:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202448.388778:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187708160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202448.388798:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202448/real 1788202448] req@ffff8e8289539f80 x1875062187707648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202448.388817:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187707648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202448.388832:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202448.388835:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202448.388838:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187707648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202448.388844:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187707776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202448.388862:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202448.388870:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202448/real 1788202448] req@ffff8e8289538380 x1875062187707776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202448.388879:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187707776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202448.388884:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202448.388887:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202448.388889:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187707776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202448.388912:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187707904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202448.388924:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202448.388931:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202448/real 1788202448] req@ffff8e828953bb80 x1875062187707904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202448.388946:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187707904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202448.388952:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202448.388954:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202448.388956:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187707904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202448.388961:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187708032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202448.388974:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202448.388985:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202448/real 1788202448] req@ffff8e828953b100 x1875062187708032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202448.388993:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187708032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202448.388999:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202448.389001:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202448.389003:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187708032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202448.389006:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187708160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202448.389017:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202448.389023:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202448/real 1788202448] req@ffff8e8289539500 x1875062187708160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202448.389029:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187708160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202448.389035:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202448.389037:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202448.389039:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187708160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202449.092451:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202450.116550:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202451.140510:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202452.164532:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202453.188544:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202453.508421:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202453.508432:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202453.508436:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202453.508447:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3908 00000100:00080000:2.0:1788202453.508459:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202453.508467:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202453.508505:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187708288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202453.508534:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202453.508538:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202453.508540:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202453.508546:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3908 00000100:00080000:2.0:1788202453.508550:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202453.508556:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202453.508564:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202453.508577:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187708416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202453.508584:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187708288:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202453.508589:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202453.508592:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202453.508594:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202453.508597:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3908 00000100:00080000:2.0:1788202453.508600:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202453.508603:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202453.508606:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202453.508616:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187708544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202453.508633:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202453.508637:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202453.508639:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788202453.508645:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788202453.508648:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3908 00000100:00080000:2.0:1788202453.508655:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202453.508661:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202453.508667:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202453.508682:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187708672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202453.508685:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202453/real 1788202453] req@ffff8e82894e5880 x1875062187708288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202453.508713:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202453.508716:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202453.508720:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788202453.508721:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187708288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202453.508728:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3908 00000100:00080000:2.0:1788202453.508734:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202453.508738:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202453.508740:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:1.0:1788202453.508744:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000008:00100000:2.0:1788202453.508747:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202453.508749:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187708288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202453.508762:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187708800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202453.508773:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187708416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202453.508790:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202453.508799:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202453/real 1788202453] req@ffff8e82894e4000 x1875062187708416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202453.508807:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187708416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202453.508813:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202453.508815:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202453.508817:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187708416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202453.508821:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187708544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202453.508830:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202453.508836:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202453/real 1788202453] req@ffff8e82894e6a00 x1875062187708544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202453.508848:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187708544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202453.508853:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202453.508855:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202453.508856:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187708544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202453.508860:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187708672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202453.508869:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202453.508875:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202453/real 1788202453] req@ffff8e82894e6680 x1875062187708672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202453.508882:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187708672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202453.508887:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202453.508889:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202453.508890:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187708672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202453.508899:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187708800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202453.508909:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202453.508914:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202453/real 1788202453] req@ffff8e82894e7480 x1875062187708800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202453.508921:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187708800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202453.508930:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202453.508932:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202453.508934:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187708800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202454.212467:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202455.236485:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202456.260454:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202457.284504:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202458.308449:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202458.628455:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202458.628464:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202458.628468:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202458.628476:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3913 00000100:00080000:0.0:1788202458.628485:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202458.628491:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202458.628524:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187708928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202458.628551:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202458.628555:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202458.628556:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202458.628560:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3913 00000100:00080000:0.0:1788202458.628565:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202458.628569:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202458.628578:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202458.628591:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187709056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202458.628603:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202458.628606:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202458.628607:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202458.628611:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3913 00000100:00080000:0.0:1788202458.628614:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202458.628617:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202458.628621:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202458.628621:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187708928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202458.628631:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187709184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202458.628640:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202458.628643:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202458.628644:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202458.628648:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3913 00000100:00080000:0.0:1788202458.628650:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202458.628653:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202458.628657:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202458.628666:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187709312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202458.628673:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202458.628677:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1788202458.628678:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202458.628679:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202458.628682:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3913 00000100:00080000:0.0:1788202458.628685:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202458.628688:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202458.628692:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202458.628702:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187709440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202458.628718:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202458/real 1788202458] req@ffff8e828793c000 x1875062187708928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202458.628736:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187708928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202458.628746:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202458.628750:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202458.628754:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187708928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202458.628761:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187709056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202458.628780:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202458.628790:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202458/real 1788202458] req@ffff8e828793dc00 x1875062187709056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202458.628802:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187709056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202458.628811:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202458.628814:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202458.628817:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187709056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202458.628842:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187709184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202458.628863:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202458.628872:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202458/real 1788202458] req@ffff8e828793f480 x1875062187709184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202458.628880:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187709184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202458.628886:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202458.628888:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202458.628890:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187709184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202458.628893:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187709312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202458.628906:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202458.628912:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202458/real 1788202458] req@ffff8e828793c700 x1875062187709312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202458.628919:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187709312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202458.628924:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202458.628925:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202458.628928:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187709312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202458.628931:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187709440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202458.628941:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202458.628950:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202458/real 1788202458] req@ffff8e828793ca80 x1875062187709440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202458.628962:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187709440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202458.628970:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202458.628973:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202458.628977:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187709440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202459.332478:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202460.356451:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202461.380492:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202462.404561:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202463.428496:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202463.748473:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202463.748481:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202463.748486:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202463.748495:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3918 00000100:00080000:0.0:1788202463.748503:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202463.748510:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202463.748543:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187709568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202463.748577:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202463.748581:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202463.748583:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202463.748586:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3918 00000100:00080000:0.0:1788202463.748588:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202463.748591:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202463.748599:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202463.748612:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187709696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202463.748616:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187709568:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202463.748622:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202463.748628:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202463.748629:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202463.748634:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3918 00000100:00080000:0.0:1788202463.748638:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202463.748642:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202463.748646:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202463.748657:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187709824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202463.748672:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202463.748679:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202463.748682:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202463.748690:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3918 00000100:00080000:0.0:1788202463.748695:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788202463.748697:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202463.748704:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202463.748713:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202463.748727:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187709952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202463.748743:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202463.748747:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202463.748750:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202463.748755:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3918 00000100:00080000:0.0:1788202463.748760:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202463.748762:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202463/real 1788202463] req@ffff8e828793d500 x1875062187709568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202463.748768:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202463.748778:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202463.748792:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187709568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202463.748793:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187710080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202463.748835:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202463.748842:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202463.748849:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187709568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202463.748879:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187709696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202463.748914:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202463.748938:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202463/real 1788202463] req@ffff8e828793d880 x1875062187709696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202463.748952:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187709696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202463.748961:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202463.748965:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202463.748969:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187709696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202463.748976:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187709824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202463.748998:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202463.749009:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202463/real 1788202463] req@ffff8e828793c380 x1875062187709824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202463.749022:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187709824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202463.749030:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202463.749034:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202463.749039:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187709824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202463.749058:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187709952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202463.749084:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202463.749106:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202463/real 1788202463] req@ffff8e828793fb80 x1875062187709952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202463.749137:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187709952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202463.749147:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202463.749151:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202463.749156:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187709952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202463.749163:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187710080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202463.749195:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202463.749217:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202463/real 1788202463] req@ffff8e828793ca80 x1875062187710080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202463.749228:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187710080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202463.749236:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202463.749240:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202463.749243:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187710080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202464.452486:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202465.476555:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202466.500532:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202467.524518:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202468.548521:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202468.868418:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202468.868433:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202468.868439:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202468.868453:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3923 00000100:00080000:0.0:1788202468.868467:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202468.868476:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202468.868524:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187710208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202468.868560:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202468.868565:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202468.868567:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202468.868572:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3923 00000100:00080000:0.0:1788202468.868577:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202468.868582:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202468.868594:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202468.868608:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187710336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202468.868613:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187710208:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202468.868628:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202468.868632:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202468.868634:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202468.868639:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3923 00000100:00080000:0.0:1788202468.868643:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202468.868647:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202468.868653:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202468.868665:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187710464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202468.868676:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202468.868680:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202468.868682:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202468.868686:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3923 00000100:00080000:0.0:1788202468.868692:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202468.868696:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202468.868702:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788202468.868705:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202468.868717:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187710592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202468.868728:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202468.868732:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202468.868734:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202468.868738:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3923 00000100:00080000:0.0:1788202468.868743:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202468.868747:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202468.868753:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202468.868754:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202468/real 1788202468] req@ffff8e828793c700 x1875062187710208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202468.868770:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187710720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202468.868779:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187710208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202468.868818:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202468.868825:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202468.868829:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187710208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202468.868856:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187710336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202468.868882:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202468.868897:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202468/real 1788202468] req@ffff8e828793f480 x1875062187710336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202468.868912:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187710336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202468.868928:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202468.868931:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202468.868934:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187710336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202468.868939:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187710464:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202468.868956:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202468.868965:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202468/real 1788202468] req@ffff8e828793dc00 x1875062187710464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202468.868973:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187710464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202468.868984:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202468.868986:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202468.868989:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187710464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202468.868992:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187710592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202468.869004:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202468.869013:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202468/real 1788202468] req@ffff8e828793c000 x1875062187710592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202468.869043:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187710592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202468.869054:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202468.869058:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202468.869062:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187710592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202468.869069:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187710720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202468.869088:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202468.869097:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202468/real 1788202468] req@ffff8e828793e300 x1875062187710720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202468.869105:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187710720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202468.869110:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202468.869112:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202468.869115:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187710720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202469.572496:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202470.596544:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202471.620489:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202472.644551:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202473.668496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202473.988482:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202473.988493:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202473.988498:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202473.988510:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3929 00000100:00080000:2.0:1788202473.988522:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202473.988538:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202473.988584:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187710848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202473.988610:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202473.988614:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202473.988615:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202473.988619:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3929 00000100:00080000:2.0:1788202473.988621:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202473.988624:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202473.988635:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202473.988645:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187710976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202473.988653:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202473.988656:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202473.988657:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202473.988660:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3929 00000100:00080000:2.0:1788202473.988662:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202473.988664:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202473.988668:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202473.988672:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187710848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202473.988678:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187711104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202473.988686:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202473.988688:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202473.988690:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202473.988692:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3929 00000100:00080000:2.0:1788202473.988696:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202473.988704:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202473.988708:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202473.988718:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187711232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202473.988724:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202473.988727:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202473.988729:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202473.988732:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3929 00000100:00080000:2.0:1788202473.988735:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202473.988739:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202473.988742:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788202473.988751:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202473.988754:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187711360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202473.988801:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202473/real 1788202473] req@ffff8e82894e4e00 x1875062187710848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202473.988816:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187710848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202473.988825:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202473.988828:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202473.988832:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187710848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202473.988837:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187710976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202473.988853:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202473.988862:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202473/real 1788202473] req@ffff8e82894e7800 x1875062187710976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202473.988870:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187710976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202473.988876:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202473.988878:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202473.988881:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187710976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202473.988904:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187711104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202473.988916:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202473.988922:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202473/real 1788202473] req@ffff8e82894e6300 x1875062187711104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202473.988930:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187711104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202473.988935:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202473.988937:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202473.988939:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187711104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202473.988960:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187711232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202473.988977:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202473.988985:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202473/real 1788202473] req@ffff8e82894e4a80 x1875062187711232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202473.988993:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187711232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202473.988999:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202473.989002:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202473.989006:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187711232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202473.989013:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187711360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202473.989034:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202473.989046:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202473/real 1788202473] req@ffff8e82894e7100 x1875062187711360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202473.989059:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187711360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202473.989068:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202473.989071:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202473.989075:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187711360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202474.692573:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202475.716483:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202476.740520:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202477.764524:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202478.788518:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202479.108488:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202479.108502:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202479.108508:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202479.108521:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3934 00000100:00080000:0.0:1788202479.108536:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202479.108553:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202479.108604:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187711488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202479.108640:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202479.108644:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202479.108646:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202479.108651:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3934 00000100:00080000:0.0:1788202479.108655:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202479.108659:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202479.108671:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202479.108685:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187711616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202479.108696:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202479.108700:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202479.108702:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202479.108706:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3934 00000100:00080000:0.0:1788202479.108709:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202479.108714:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202479.108720:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202479.108725:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187711488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202479.108735:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187711744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202479.108746:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202479.108750:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202479.108751:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202479.108755:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3934 00000100:00080000:0.0:1788202479.108759:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202479.108763:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202479.108769:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202479.108781:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187711872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202479.108791:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202479.108794:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202479.108796:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202479.108800:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3934 00000100:00080000:0.0:1788202479.108803:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788202479.108805:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202479.108810:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202479.108816:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202479.108828:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187712000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202479.108881:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202479/real 1788202479] req@ffff8e828793ce00 x1875062187711488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202479.108902:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187711488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202479.108914:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202479.108918:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202479.108923:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187711488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202479.108930:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187711616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202479.108951:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202479.108962:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202479/real 1788202479] req@ffff8e828793f800 x1875062187711616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202479.108969:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187711616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202479.108975:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202479.108977:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202479.108979:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187711616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202479.109000:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187711744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202479.109011:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202479.109017:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202479/real 1788202479] req@ffff8e828793e680 x1875062187711744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202479.109023:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187711744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202479.109028:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202479.109029:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202479.109031:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187711744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202479.109033:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187711872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202479.109043:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202479.109048:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202479/real 1788202479] req@ffff8e828793f100 x1875062187711872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202479.109054:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187711872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202479.109060:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202479.109062:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202479.109063:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187711872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202479.109066:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187712000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202479.109074:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202479.109079:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202479/real 1788202479] req@ffff8e828793e300 x1875062187712000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202504 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202479.109084:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187712000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202504 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202479.109088:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202479.109090:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202479.109092:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187712000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202479.812454:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202480.836511:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202481.860477:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202482.884426:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202483.908470:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202484.228413:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202484.228420:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202484.228424:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202484.228432:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3939 00000100:00080000:2.0:1788202484.228440:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202484.228446:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202484.228476:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187712128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202484.228495:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202484.228497:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202484.228498:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202484.228501:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3939 00000100:00080000:2.0:1788202484.228503:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202484.228505:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202484.228512:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202484.228520:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187712256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202484.228535:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187712128:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202484.228535:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202484.228543:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202484.228545:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202484.228550:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3939 00000100:00080000:2.0:1788202484.228554:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202484.228558:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202484.228564:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202484.228576:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187712384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202484.228582:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202484.228585:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202484.228586:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202484.228588:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3939 00000800:00000100:1.0:1788202484.228592:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788202484.228593:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202484.228598:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202484.228601:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202484.228609:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187712512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202484.228615:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202484.228617:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202484.228618:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202484.228620:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3939 00000100:00080000:2.0:1788202484.228623:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202484.228628:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202484/real 1788202484] req@ffff8e82894e6d80 x1875062187712128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202484.228628:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202484.228630:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202484.228638:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187712640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202484.228668:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187712128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202484.228677:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202484.228681:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202484.228683:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187712128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202484.228699:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187712256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202484.228711:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202484.228719:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202484/real 1788202484] req@ffff8e82894e5f80 x1875062187712256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202484.228726:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187712256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202484.228730:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202484.228732:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202484.228734:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187712256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202484.228737:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187712384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202484.228745:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202484.228750:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202484/real 1788202484] req@ffff8e82894e4380 x1875062187712384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202484.228756:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187712384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202484.228761:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202484.228763:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202484.228764:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187712384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202484.228788:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187712512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202484.228800:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202484.228806:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202484/real 1788202484] req@ffff8e82894e5c00 x1875062187712512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202484.228812:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187712512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202484.228817:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202484.228818:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202484.228820:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187712512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202484.228823:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187712640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202484.228831:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202484.228836:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202484/real 1788202484] req@ffff8e82894e7100 x1875062187712640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202484.228842:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187712640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202509 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202484.228846:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202484.228848:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202484.228849:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187712640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202484.932499:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202485.956459:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202486.980451:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202488.004444:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202489.028496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202489.348402:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202489.348411:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202489.348414:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202489.348424:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3944 00000100:00080000:2.0:1788202489.348433:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202489.348440:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202489.348476:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187712768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202489.348503:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202489.348507:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202489.348509:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202489.348513:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3944 00000100:00080000:2.0:1788202489.348516:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202489.348520:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202489.348528:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202489.348540:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187712896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202489.348550:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202489.348556:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202489.348557:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202489.348561:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3944 00000100:00080000:2.0:1788202489.348564:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202489.348568:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202489.348574:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202489.348582:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187712768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202489.348590:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187713024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202489.348604:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202489.348607:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202489.348609:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202489.348613:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3944 00000100:00080000:2.0:1788202489.348617:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202489.348622:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202489.348626:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202489.348638:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187713152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202489.348648:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202489.348652:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1788202489.348655:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788202489.348657:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202489.348663:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3944 00000100:00080000:2.0:1788202489.348668:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202489.348674:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202489.348694:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202489.348709:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187713280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202489.348722:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202489/real 1788202489] req@ffff8e82894e4a80 x1875062187712768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202489.348737:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187712768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202489.348747:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202489.348751:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202489.348754:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187712768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202489.348760:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187712896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202489.348777:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202489.348788:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202489/real 1788202489] req@ffff8e82894e6300 x1875062187712896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202489.348799:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187712896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202489.348807:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202489.348810:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202489.348813:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187712896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202489.348839:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187713024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202489.348855:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202489.348869:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202489/real 1788202489] req@ffff8e82894e7800 x1875062187713024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202489.348877:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187713024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202489.348882:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202489.348884:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202489.348886:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187713024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202489.348889:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187713152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202489.348901:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202489.348912:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202489/real 1788202489] req@ffff8e82894e4e00 x1875062187713152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202489.348920:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187713152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202489.348925:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202489.348927:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202489.348928:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187713152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202489.348932:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187713280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202489.348946:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202489.348955:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202489/real 1788202489] req@ffff8e82894e7480 x1875062187713280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202489.348974:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187713280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202489.348987:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202489.348989:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202489.348991:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187713280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202490.052518:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202491.076530:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202492.100505:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202493.124464:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202494.148552:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202494.468443:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202494.468455:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202494.468461:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202494.468475:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3949 00000100:00080000:0.0:1788202494.468488:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202494.468497:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202494.468547:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187713408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202494.468576:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202494.468580:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202494.468582:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202494.468601:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3949 00000100:00080000:0.0:1788202494.468612:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202494.468618:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202494.468630:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202494.468638:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187713408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202494.468648:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187713536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202494.468661:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202494.468663:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202494.468665:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202494.468668:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3949 00000100:00080000:0.0:1788202494.468671:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202494.468673:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202494.468677:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202494.468686:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187713664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202494.468698:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202494.468700:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202494.468702:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202494.468705:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3949 00000100:00080000:0.0:1788202494.468721:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202494.468724:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202494.468727:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788202494.468734:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202494.468737:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187713792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202494.468745:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202494.468748:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202494.468750:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202494.468753:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3949 00000100:00080000:0.0:1788202494.468759:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202494.468762:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202494.468765:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202494.468774:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187713920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202494.468785:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202494/real 1788202494] req@ffff8e828793c000 x1875062187713408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202494.468802:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187713408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202494.468811:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202494.468815:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202494.468818:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187713408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202494.468840:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187713536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202494.468857:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202494.468866:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202494/real 1788202494] req@ffff8e828793dc00 x1875062187713536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202494.468874:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187713536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202494.468880:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202494.468883:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202494.468885:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187713536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202494.468889:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187713664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202494.468899:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202494.468906:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202494/real 1788202494] req@ffff8e828793f480 x1875062187713664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202494.468913:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187713664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202494.468926:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202494.468930:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202494.468933:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187713664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202494.468940:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187713792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202494.468962:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202494.468976:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202494/real 1788202494] req@ffff8e828793c700 x1875062187713792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202494.468989:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187713792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202494.468998:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202494.469002:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202494.469006:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187713792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202494.469013:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187713920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202494.469035:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202494.469047:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202494/real 1788202494] req@ffff8e828793ca80 x1875062187713920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202494.469060:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187713920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202494.469070:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202494.469073:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202494.469077:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187713920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202495.172484:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202496.196537:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202497.220465:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202498.244515:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202499.268529:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202499.588407:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202499.588420:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202499.588425:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202499.588437:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3954 00000100:00080000:0.0:1788202499.588451:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202499.588460:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202499.588527:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187714048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202499.588555:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202499.588558:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202499.588559:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202499.588563:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3954 00000100:00080000:0.0:1788202499.588567:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202499.588571:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202499.588581:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202499.588591:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187714048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202499.588599:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187714176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202499.588614:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202499.588619:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202499.588621:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202499.588627:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3954 00000100:00080000:0.0:1788202499.588633:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202499.588638:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202499.588646:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202499.588660:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187714304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202499.588669:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788202499.588675:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202499.588676:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202499.588680:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202499.588684:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3954 00000100:00080000:0.0:1788202499.588688:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202499.588691:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202499.588696:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202499.588706:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187714432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202499.588712:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202499.588715:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202499.588716:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202499.588719:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3954 00000100:00100000:2.0:1788202499.588723:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202499/real 1788202499] req@ffff8e828793fb80 x1875062187714048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202499.588724:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202499.588732:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202499.588739:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202499.588751:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187714560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202499.588753:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187714048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202499.588779:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202499.588783:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202499.588787:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187714048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202499.588817:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187714176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202499.588834:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202499.588844:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202499/real 1788202499] req@ffff8e828793c380 x1875062187714176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202499.588853:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187714176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202499.588860:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202499.588862:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202499.588864:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187714176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202499.588868:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187714304:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202499.588882:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202499.588889:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202499/real 1788202499] req@ffff8e828793d880 x1875062187714304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202499.588898:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187714304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202499.588904:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202499.588906:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202499.588908:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187714304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202499.588912:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187714432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202499.588926:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202499.588934:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202499/real 1788202499] req@ffff8e828793d500 x1875062187714432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202499.588957:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187714432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202499.588966:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202499.588970:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202499.588975:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187714432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202499.588985:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187714560:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202499.589012:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202499.589034:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202499/real 1788202499] req@ffff8e828793ca80 x1875062187714560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202499.589052:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187714560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202499.589058:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202499.589060:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202499.589063:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187714560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202500.292548:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202501.316555:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202502.340534:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202503.364557:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202504.388521:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202504.708473:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202504.708484:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202504.708489:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202504.708501:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3959 00000100:00080000:2.0:1788202504.708513:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202504.708521:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202504.708565:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187714688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202504.708594:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202504.708599:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202504.708601:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202504.708606:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3959 00000100:00080000:2.0:1788202504.708610:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202504.708616:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202504.708636:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202504.708653:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187714816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202504.708666:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187714688:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202504.708683:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202504.708685:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202504.708687:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202504.708691:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3959 00000100:00080000:2.0:1788202504.708695:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202504.708699:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202504.708705:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202504.708719:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187714944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202504.708730:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202504.708734:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202504.708736:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202504.708741:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3959 00000100:00080000:2.0:1788202504.708746:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202504.708751:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202504.708756:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202504.708766:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187715072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202504.708773:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202504.708776:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202504.708778:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202504.708780:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3959 00000100:00080000:2.0:1788202504.708784:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202504.708787:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202504.708797:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202504.708808:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187715200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202504.708822:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187714816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202504.708856:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187714944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202504.708877:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187715072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202504.708901:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187715200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202505.412606:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202506.436528:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202507.460571:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788202507.780676:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788202507.780739:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788202507.780761:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202507.780817:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202507.780829:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202507.780838:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202507.780847:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788202507.781009:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202504/real 1788202507] req@ffff8e82894e7480 x1875062187715200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202507.781047:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187715200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202507.781061:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202507.781068:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202507.781073:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187715200:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202507.781083:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202504/real 1788202507] req@ffff8e82894e6a00 x1875062187714816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202507.781096:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187714816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202507.781106:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202507.781110:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202507.781114:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187714816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202507.781122:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202504/real 1788202507] req@ffff8e82894e4000 x1875062187714944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202507.781133:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187714944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202507.781142:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202507.781146:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202507.781150:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187714944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202507.781158:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202504/real 1788202507] req@ffff8e82894e5880 x1875062187715072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202507.781170:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187715072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202507.781178:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202507.781183:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202507.781186:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187715072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202507.781193:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202504/real 1788202507] req@ffff8e82894e6680 x1875062187714688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202507.781206:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187714688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202507.781214:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202507.781218:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202507.781221:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187714688:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788202508.484540:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202509.508556:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202509.828418:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202509.828426:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202509.828430:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202509.828439:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3964 00000100:00080000:0.0:1788202509.828457:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202509.828465:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202509.828502:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187715328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202509.828533:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202509.828537:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202509.828539:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202509.828544:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3964 00000100:00080000:0.0:1788202509.828548:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202509.828553:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202509.828562:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202509.828576:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187715456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202509.828586:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202509.828590:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202509.828592:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202509.828596:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3964 00000100:00080000:0.0:1788202509.828599:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202509.828603:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202509.828608:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202509.828616:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187715584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202509.828625:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202509.828628:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202509.828630:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202509.828633:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3964 00000100:00080000:0.0:1788202509.828636:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202509.828656:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202509.828662:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202509.828675:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187715712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202509.828684:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202509.828687:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202509.828688:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202509.828692:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3964 00000100:00080000:0.0:1788202509.828695:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202509.828699:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202509.828703:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202509.828713:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187715840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202509.829217:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187715328:192.168.201.121@tcp:250:kworker.0 00000800:00000100:1.0:1788202509.829282:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202509.829378:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202509/real 1788202509] req@ffff8e828793c700 x1875062187715328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202509.829399:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187715328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202509.829411:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202509.829415:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202509.829419:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187715328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202509.829429:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187715456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202509.829453:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202509.829466:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202509/real 1788202509] req@ffff8e828793f480 x1875062187715456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202509.829478:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187715456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202509.829487:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202509.829490:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202509.829494:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187715456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202509.829499:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187715584:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202509.829519:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202509.829531:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202509/real 1788202509] req@ffff8e828793dc00 x1875062187715584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202509.829542:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187715584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202509.829556:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202509.829561:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202509.829564:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187715584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202509.829570:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187715712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202509.829590:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202509.829601:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202509/real 1788202509] req@ffff8e828793c000 x1875062187715712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202509.829613:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187715712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202509.829621:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202509.829624:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202509.829627:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187715712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202509.829633:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187715840:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202509.829654:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202509.829664:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202509/real 1788202509] req@ffff8e828793e300 x1875062187715840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202509.829674:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187715840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202509.829682:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202509.829684:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202509.829688:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187715840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202510.532501:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202511.556483:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202512.580438:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202513.604449:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202514.628455:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202514.948436:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202514.948451:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202514.948457:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202514.948471:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3970 00000100:00080000:0.0:1788202514.948485:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202514.948495:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202514.948556:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187715968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202514.948591:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202514.948599:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202514.948601:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202514.948606:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3970 00000100:00080000:0.0:1788202514.948611:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202514.948615:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202514.948625:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202514.948638:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187716096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202514.948647:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202514.948651:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788202514.948653:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187715968:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202514.948654:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202514.948660:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3970 00000100:00080000:0.0:1788202514.948665:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202514.948667:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202514.948670:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202514.948678:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187716224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202514.948684:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202514.948687:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202514.948688:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202514.948691:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3970 00000100:00080000:0.0:1788202514.948694:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202514.948696:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202514.948701:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202514.948708:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187716352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202514.948714:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202514.948717:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202514.948718:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202514.948721:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3970 00000100:00080000:0.0:1788202514.948724:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202514.948726:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202514.948728:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788202514.948736:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202514.948738:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187716480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202514.948794:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202514/real 1788202514] req@ffff8e828793f100 x1875062187715968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202514.948812:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187715968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202514.948821:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202514.948825:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202514.948828:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187715968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202514.948854:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187716096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202514.948879:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202514.948895:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202514/real 1788202514] req@ffff8e828793e680 x1875062187716096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202514.948907:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187716096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202514.948913:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202514.948916:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202514.948919:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187716096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202514.948924:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187716224:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202514.948940:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202514.948949:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202514/real 1788202514] req@ffff8e828793f800 x1875062187716224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202514.948962:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187716224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202514.948971:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202514.948974:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202514.948978:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187716224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202514.948985:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187716352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202514.949006:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202514.949014:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202514/real 1788202514] req@ffff8e828793ce00 x1875062187716352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202514.949023:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187716352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202514.949028:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202514.949031:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202514.949033:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187716352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202514.949037:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187716480:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202514.949049:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202514.949055:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202514/real 1788202514] req@ffff8e828793e300 x1875062187716480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202514.949066:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187716480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202514.949074:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202514.949078:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202514.949081:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187716480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202515.652461:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202516.676541:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202517.700518:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202518.724518:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202519.748501:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202520.068456:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202520.068471:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202520.068476:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202520.068486:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3975 00000100:00080000:2.0:1788202520.068499:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202520.068506:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202520.068547:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187716608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202520.068570:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202520.068573:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202520.068574:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202520.068577:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3975 00000100:00080000:2.0:1788202520.068580:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202520.068582:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202520.068591:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202520.068600:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187716736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202520.068607:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788202520.068613:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187716608:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202520.068613:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202520.068616:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202520.068623:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3975 00000100:00080000:2.0:1788202520.068625:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202520.068627:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202520.068630:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202520.068638:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187716864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202520.068644:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202520.068646:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202520.068647:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202520.068650:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3975 00000100:00080000:2.0:1788202520.068652:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202520.068655:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202520.068657:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202520.068664:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187716992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202520.068669:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202520.068671:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202520.068672:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202520.068674:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3975 00000100:00080000:2.0:1788202520.068676:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202520.068678:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202520.068681:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788202520.068687:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202520.068690:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187717120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202520.068733:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202520/real 1788202520] req@ffff8e82894e4e00 x1875062187716608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202520.068756:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187716608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202520.068764:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202520.068767:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202520.068770:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187716608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202520.068790:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187716736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202520.068804:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202520.068812:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202520/real 1788202520] req@ffff8e82894e7800 x1875062187716736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202520.068820:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187716736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202520.068825:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202520.068827:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202520.068829:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187716736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202520.068832:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187716864:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202520.068842:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202520.068848:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202520/real 1788202520] req@ffff8e82894e6300 x1875062187716864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202520.068859:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187716864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202520.068867:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202520.068870:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202520.068874:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187716864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202520.068880:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187716992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202520.068900:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202520.068909:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202520/real 1788202520] req@ffff8e82894e4a80 x1875062187716992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202520.068920:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187716992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202520.068928:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202520.068930:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202520.068932:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187716992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202520.068936:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187717120:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202520.068948:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202520.068954:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202520/real 1788202520] req@ffff8e82894e7100 x1875062187717120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202545 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202520.068960:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187717120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202545 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202520.068965:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202520.068967:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202520.068969:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187717120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202520.772516:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202521.796537:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202522.820504:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202523.844546:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202524.868453:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202525.188434:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202525.188445:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202525.188449:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202525.188461:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3980 00000100:00080000:0.0:1788202525.188472:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202525.188479:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202525.188520:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187717248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202525.188543:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202525.188546:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202525.188547:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202525.188551:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3980 00000100:00080000:0.0:1788202525.188554:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202525.188556:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202525.188566:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202525.188574:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187717376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202525.188581:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202525.188584:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202525.188585:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202525.188587:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3980 00000100:00080000:0.0:1788202525.188589:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202525.188591:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202525.188593:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202525.188601:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187717504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202525.188603:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187717248:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202525.188608:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202525.188614:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202525.188615:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202525.188617:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3980 00000100:00080000:0.0:1788202525.188619:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202525.188621:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202525.188624:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202525.188631:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187717632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202525.188641:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202525.188643:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202525.188645:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202525.188647:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3980 00000100:00080000:0.0:1788202525.188649:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202525.188651:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202525.188653:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202525.188658:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187717760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202525.188703:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202525.188758:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202525/real 1788202525] req@ffff8e828793c000 x1875062187717248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202525.188778:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187717248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202525.188798:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202525.188804:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202525.188808:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187717248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202525.188817:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187717376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202525.188841:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202525.188854:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202525/real 1788202525] req@ffff8e828793dc00 x1875062187717376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202525.188866:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187717376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202525.188874:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202525.188877:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202525.188880:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187717376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202525.188909:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187717504:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202525.188930:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202525.188941:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202525/real 1788202525] req@ffff8e828793f480 x1875062187717504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202525.188961:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187717504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202525.188969:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202525.188972:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202525.188975:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187717504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202525.188981:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187717632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202525.188999:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202525.189009:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202525/real 1788202525] req@ffff8e828793c700 x1875062187717632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202525.189019:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187717632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202525.189027:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202525.189030:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202525.189033:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187717632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202525.189038:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187717760:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202525.189079:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202525.189090:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202525/real 1788202525] req@ffff8e828793ca80 x1875062187717760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202550 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202525.189101:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187717760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202550 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202525.189109:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202525.189112:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202525.189115:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187717760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202525.892499:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202526.916519:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202527.940523:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202528.964502:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202529.988465:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202530.308450:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202530.308461:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202530.308466:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202530.308477:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3985 00000100:00080000:1.0:1788202530.308490:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202530.308497:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202530.308544:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187717888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202530.308574:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202530.308577:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202530.308579:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202530.308582:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3985 00000100:00080000:1.0:1788202530.308584:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202530.308586:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202530.308595:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202530.308603:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187718016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202530.308609:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202530.308612:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202530.308614:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202530.308616:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3985 00000100:00080000:1.0:1788202530.308618:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202530.308620:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202530.308623:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202530.308629:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187718144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202530.308635:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187717888:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202530.308639:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202530.308642:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202530.308644:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202530.308646:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3985 00000100:00080000:1.0:1788202530.308648:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202530.308650:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202530.308653:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202530.308659:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187718272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202530.308665:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202530.308667:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202530.308668:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202530.308670:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3985 00000100:00080000:1.0:1788202530.308672:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202530.308674:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202530.308677:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202530.308683:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187718400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202530.308719:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202530.308769:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202530/real 1788202530] req@ffff8e828d720a80 x1875062187717888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202530.308784:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187717888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202530.308793:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202530.308796:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202530.308799:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187717888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202530.308803:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187718016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202530.308818:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202530.308826:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202530/real 1788202530] req@ffff8e828d721180 x1875062187718016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202530.308833:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187718016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202530.308838:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202530.308840:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202530.308841:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187718016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202530.308863:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187718144:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202530.308873:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202530.308879:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202530/real 1788202530] req@ffff8e828d722d80 x1875062187718144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202530.308886:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187718144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202530.308891:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202530.308893:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202530.308895:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187718144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202530.308897:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187718272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202530.308907:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202530.308912:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202530/real 1788202530] req@ffff8e828d722300 x1875062187718272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202530.308918:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187718272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202530.308923:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202530.308924:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202530.308926:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187718272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202530.308929:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187718400:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202530.308937:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202530.308942:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202530/real 1788202530] req@ffff8e828d720380 x1875062187718400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202530.308952:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187718400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202530.308957:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202530.308958:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202530.308960:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187718400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202531.012560:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202532.036507:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202533.060481:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202534.084458:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202535.108500:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202535.428434:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202535.428445:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202535.428450:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202535.428461:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3990 00000100:00080000:0.0:1788202535.428472:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202535.428479:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202535.428513:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187718528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202535.428554:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202535.428558:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202535.428560:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202535.428566:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3990 00000100:00080000:0.0:1788202535.428570:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202535.428579:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202535.428589:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202535.428597:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187718528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202535.428607:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187718656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202535.428620:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202535.428624:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202535.428626:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202535.428631:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3990 00000100:00080000:0.0:1788202535.428635:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202535.428640:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202535.428645:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202535.428659:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187718784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202535.428668:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202535.428674:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1788202535.428674:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202535.428676:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202535.428681:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3990 00000100:00080000:0.0:1788202535.428685:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202535.428688:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202535.428693:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202535.428701:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187718912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202535.428707:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202535.428710:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202535.428712:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202535.428716:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3990 00000100:00080000:0.0:1788202535.428718:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202535.428721:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202535.428723:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202535.428730:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187719040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202535.428893:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202535/real 1788202535] req@ffff8e828793d500 x1875062187718528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202535.428913:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187718528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202535.428922:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202535.428926:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202535.428929:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187718528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202535.428952:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187718656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202535.428973:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202535.429022:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202535/real 1788202535] req@ffff8e828793d880 x1875062187718656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202535.429031:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187718656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202535.429037:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202535.429041:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202535.429045:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187718656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202535.429051:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187718784:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202535.429072:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202535.429084:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202535/real 1788202535] req@ffff8e828793c380 x1875062187718784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202535.429102:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187718784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202535.429107:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202535.429110:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202535.429112:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187718784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202535.429116:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187718912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202535.429131:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202535.429148:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202535/real 1788202535] req@ffff8e828793fb80 x1875062187718912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202535.429156:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187718912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202535.429161:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202535.429164:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202535.429166:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187718912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202535.429169:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187719040:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202535.429180:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202535.429187:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202535/real 1788202535] req@ffff8e828793ca80 x1875062187719040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202535.429194:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187719040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202535.429213:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202535.429217:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202535.429220:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187719040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202536.132436:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202537.156533:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202538.180505:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202539.204549:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202540.228522:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202540.548423:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202540.548433:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202540.548438:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202540.548450:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 3995 00000100:00080000:0.0:1788202540.548462:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202540.548486:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202540.548539:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187719168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202540.548570:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202540.548573:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202540.548575:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202540.548579:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3995 00000100:00080000:0.0:1788202540.548583:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202540.548588:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202540.548598:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202540.548613:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187719296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202540.548620:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187719168:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202540.548671:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202540.548674:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202540.548676:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202540.548681:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3995 00000100:00080000:0.0:1788202540.548685:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202540.548688:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202540.548693:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202540.548720:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187719424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202540.548722:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202540.548735:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202540.548739:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202540.548741:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202540.548747:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3995 00000100:00080000:0.0:1788202540.548750:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202540.548753:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202540.548759:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202540.548767:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202540/real 1788202540] req@ffff8e828793c700 x1875062187719168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202540.548770:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187719552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202540.548830:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788202540.548835:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187719168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202540.548836:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202540.548841:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202540.548855:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 3995 00000100:00080000:0.0:1788202540.548865:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202540.548866:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202540.548876:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202540.548881:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187719168:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202540.548897:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202540.548906:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202540.548914:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187719296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202540.548928:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187719680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202540.548949:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202540.548963:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202540/real 1788202540] req@ffff8e828793f480 x1875062187719296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202540.548974:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187719296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202540.548981:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202540.548985:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202540.548990:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187719296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202540.548996:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187719424:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202540.549009:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202540.549015:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202540/real 1788202540] req@ffff8e828793dc00 x1875062187719424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202540.549023:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187719424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202540.549027:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202540.549029:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202540.549031:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187719424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202540.549035:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187719552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202540.549046:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202540.549051:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202540/real 1788202540] req@ffff8e828793c000 x1875062187719552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202540.549058:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187719552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202540.549063:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202540.549065:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202540.549067:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187719552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202540.549077:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187719680:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202540.549087:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202540.549093:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202540/real 1788202540] req@ffff8e828793e300 x1875062187719680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202540.549101:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187719680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202540.549116:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202540.549119:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202540.549122:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187719680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202541.252560:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202542.276540:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202543.300513:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202544.324536:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202545.348511:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202545.668450:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202545.668462:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202545.668467:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202545.668479:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4000 00000100:00080000:0.0:1788202545.668491:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202545.668499:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202545.668546:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187719808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202545.668573:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202545.668576:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202545.668578:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202545.668581:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4000 00000100:00080000:0.0:1788202545.668583:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202545.668587:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202545.668596:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202545.668606:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187719936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202545.668614:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788202545.668619:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187719808:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202545.668620:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202545.668624:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202545.668630:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4000 00000100:00080000:0.0:1788202545.668633:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202545.668635:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202545.668637:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202545.668645:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187720064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202545.668652:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202545.668655:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202545.668656:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202545.668659:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4000 00000100:00080000:0.0:1788202545.668661:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202545.668663:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202545.668666:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202545.668673:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187720192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202545.668679:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202545.668681:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202545.668682:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202545.668685:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4000 00000100:00080000:0.0:1788202545.668687:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202545.668690:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202545.668694:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788202545.668730:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202545.668737:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187720320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202545.668802:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202545/real 1788202545] req@ffff8e828793ce00 x1875062187719808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202545.668825:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187719808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202545.668839:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202545.668844:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202545.668848:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187719808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202545.668879:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187719936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202545.668905:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202545.668920:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202545/real 1788202545] req@ffff8e828793f800 x1875062187719936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202545.668933:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187719936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202545.668950:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202545.668954:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202545.668958:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187719936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202545.668976:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187720064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202545.668998:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202545.669010:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202545/real 1788202545] req@ffff8e828793e680 x1875062187720064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202545.669021:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187720064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202545.669027:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202545.669029:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202545.669031:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187720064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202545.669035:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187720192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202545.669050:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202545.669057:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202545/real 1788202545] req@ffff8e828793f100 x1875062187720192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202545.669064:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187720192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202545.669069:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202545.669071:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202545.669074:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187720192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202545.669077:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187720320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202545.669087:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202545.669092:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202545/real 1788202545] req@ffff8e828793e300 x1875062187720320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202545.669099:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187720320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202545.669104:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202545.669106:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202545.669108:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187720320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202546.372530:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202547.396552:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202548.420493:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202549.444469:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202550.468498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202550.788486:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202550.788497:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202550.788502:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202550.788512:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4005 00000100:00080000:1.0:1788202550.788525:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202550.788532:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202550.788577:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187720448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202550.788606:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202550.788630:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202550.788631:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202550.788636:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4005 00000100:00080000:1.0:1788202550.788639:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202550.788642:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202550.788653:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202550.788664:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187720576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202550.788673:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202550.788676:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202550.788677:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202550.788681:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4005 00000100:00080000:1.0:1788202550.788684:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202550.788688:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202550.788688:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187720448:192.168.201.121@tcp:250:kworker.0 00000008:00100000:1.0:1788202550.788693:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202550.788703:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187720704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202550.788710:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202550.788712:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202550.788712:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202550.788715:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4005 00000100:00080000:1.0:1788202550.788717:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202550.788719:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202550.788722:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202550.788729:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187720832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202550.788736:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202550.788739:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202550.788740:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202550.788742:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4005 00000100:00080000:1.0:1788202550.788745:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202550.788754:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202550.788757:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202550.788764:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187720960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202550.788784:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202550.788867:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202550/real 1788202550] req@ffff8e828d720e00 x1875062187720448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202550.788891:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187720448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202550.788905:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202550.788910:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202550.788915:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187720448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202550.788947:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187720576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202550.788973:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202550.788989:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202550/real 1788202550] req@ffff8e828d720000 x1875062187720576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202550.789003:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187720576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202550.789013:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202550.789017:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202550.789021:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187720576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202550.789028:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187720704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202550.789059:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202550.789073:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202550/real 1788202550] req@ffff8e828d721500 x1875062187720704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202550.789087:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187720704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202550.789096:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202550.789100:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202550.789104:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187720704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202550.789111:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187720832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202550.789134:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202550.789146:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202550/real 1788202550] req@ffff8e828d723800 x1875062187720832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202550.789159:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187720832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202550.789168:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202550.789172:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202550.789175:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187720832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202550.789182:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187720960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202550.789204:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202550.789217:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202550/real 1788202550] req@ffff8e828d722680 x1875062187720960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202550.789230:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187720960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202550.789239:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202550.789243:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202550.789247:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187720960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202551.492456:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202552.516511:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202553.540426:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202554.564508:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202555.588552:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202555.908451:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202555.908462:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202555.908467:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202555.908477:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4010 00000100:00080000:0.0:1788202555.908488:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202555.908496:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202555.908546:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187721088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202555.908582:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202555.908586:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202555.908588:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202555.908592:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4010 00000100:00080000:0.0:1788202555.908596:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202555.908600:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202555.908610:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202555.908624:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187721216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202555.908636:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202555.908640:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202555.908644:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202555.908645:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187721088:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202555.908650:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4010 00000100:00080000:0.0:1788202555.908658:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202555.908662:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202555.908667:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202555.908680:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187721344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202555.908691:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202555.908695:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202555.908697:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202555.908701:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4010 00000100:00080000:0.0:1788202555.908707:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202555.908712:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202555.908718:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788202555.908720:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202555.908735:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187721472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202555.908746:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202555.908750:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202555.908752:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202555.908756:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4010 00000100:00080000:0.0:1788202555.908760:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202555.908765:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202555.908770:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202555.908774:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202555/real 1788202555] req@ffff8e828793c000 x1875062187721088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202555.908786:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187721600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202555.908801:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187721088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202555.908817:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202555.908823:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202555.908828:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187721088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202555.908853:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187721216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202555.908877:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202555.908891:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202555/real 1788202555] req@ffff8e828793dc00 x1875062187721216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202555.908904:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187721216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202555.908921:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202555.908925:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202555.908929:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187721216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202555.908935:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187721344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202555.908955:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202555.908967:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202555/real 1788202555] req@ffff8e828793f480 x1875062187721344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202555.908980:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187721344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202555.908989:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202555.908993:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202555.908997:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187721344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202555.909004:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187721472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202555.909025:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202555.909038:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202555/real 1788202555] req@ffff8e828793c700 x1875062187721472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202555.909051:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187721472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202555.909060:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202555.909064:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202555.909068:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187721472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202555.909075:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187721600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202555.909096:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202555.909107:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202555/real 1788202555] req@ffff8e828793ca80 x1875062187721600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202555.909120:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187721600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202555.909137:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202555.909140:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202555.909144:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187721600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202556.612531:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202557.636518:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202558.660533:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202559.684588:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202560.708560:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202561.028387:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202561.028399:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202561.028404:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202561.028415:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4016 00000100:00080000:0.0:1788202561.028427:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202561.028447:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202561.028491:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187721728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202561.028517:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202561.028521:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202561.028522:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202561.028525:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4016 00000100:00080000:0.0:1788202561.028528:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202561.028532:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202561.028542:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202561.028552:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187721856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202561.028557:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187721728:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202561.028564:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202561.028567:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202561.028568:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202561.028571:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4016 00000100:00080000:0.0:1788202561.028573:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202561.028575:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202561.028580:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202561.028589:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187721984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202561.028596:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202561.028599:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202561.028600:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202561.028603:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4016 00000100:00080000:0.0:1788202561.028605:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202561.028608:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202561.028611:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202561.028617:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187722112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202561.028624:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202561.028626:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202561.028627:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202561.028630:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4016 00000100:00080000:0.0:1788202561.028632:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202561.028634:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:2.0:1788202561.028638:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788202561.028639:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202561.028652:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187722240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202561.028698:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202561/real 1788202561] req@ffff8e828793fb80 x1875062187721728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202561.028716:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187721728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202561.028727:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202561.028730:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202561.028734:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187721728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202561.028757:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187721856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202561.028773:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202561.028783:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202561/real 1788202561] req@ffff8e828793c380 x1875062187721856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202561.028791:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187721856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202561.028798:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202561.028800:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202561.028802:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187721856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202561.028806:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187721984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202561.028817:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202561.028823:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202561/real 1788202561] req@ffff8e828793d880 x1875062187721984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202561.028830:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187721984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202561.028835:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202561.028837:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202561.028839:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187721984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202561.028842:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187722112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202561.028855:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202561.028865:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202561/real 1788202561] req@ffff8e828793d500 x1875062187722112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202561.028878:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187722112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202561.028888:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202561.028891:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202561.028895:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187722112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202561.028901:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187722240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202561.028922:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202561.028933:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202561/real 1788202561] req@ffff8e828793ca80 x1875062187722240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202586 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202561.028941:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187722240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202586 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202561.028947:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202561.028949:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202561.028951:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187722240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202561.732540:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202562.756521:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202563.780567:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202564.804449:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202565.828548:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202566.148478:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202566.148487:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202566.148491:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202566.148499:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4021 00000100:00080000:2.0:1788202566.148509:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202566.148516:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202566.148557:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187722368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202566.148579:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202566.148583:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202566.148584:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202566.148588:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4021 00000100:00080000:2.0:1788202566.148590:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202566.148593:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202566.148601:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202566.148613:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187722496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202566.148625:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788202566.148630:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187722368:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202566.148640:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202566.148643:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202566.148650:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4021 00000100:00080000:2.0:1788202566.148654:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202566.148659:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202566.148665:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202566.148676:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187722624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202566.148683:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202566.148685:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202566.148687:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202566.148690:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4021 00000100:00080000:2.0:1788202566.148693:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202566.148695:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202566.148700:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788202566.148702:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202566.148709:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187722752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202566.148717:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202566.148720:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202566.148721:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202566.148724:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4021 00000100:00080000:2.0:1788202566.148726:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202566.148728:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202566.148732:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202566.148740:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187722880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202566.148769:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202566/real 1788202566] req@ffff8e82894e5c00 x1875062187722368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202566.148790:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187722368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202566.148802:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202566.148807:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202566.148812:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187722368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202566.148835:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187722496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202566.148861:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202566.148886:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202566/real 1788202566] req@ffff8e82894e4380 x1875062187722496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202566.148898:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187722496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202566.148907:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202566.148910:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202566.148914:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187722496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202566.148920:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187722624:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202566.148943:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202566.148955:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202566/real 1788202566] req@ffff8e82894e5f80 x1875062187722624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202566.148968:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187722624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202566.148981:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202566.148984:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202566.148987:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187722624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202566.148991:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187722752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202566.149008:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202566.149016:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202566/real 1788202566] req@ffff8e82894e6d80 x1875062187722752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202566.149029:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187722752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202566.149037:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202566.149040:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202566.149044:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187722752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788202566.149051:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187722880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202566.149072:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202566.149093:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202566/real 1788202566] req@ffff8e82894e7100 x1875062187722880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202591 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202566.149106:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187722880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202591 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202566.149114:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202566.149117:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202566.149120:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187722880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202566.852456:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202567.876604:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202568.900499:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202569.924538:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202570.948460:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202571.268703:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202571.268715:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202571.268721:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202571.268737:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4026 00000100:00080000:0.0:1788202571.268751:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202571.268766:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202571.268808:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187723008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202571.268842:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202571.268847:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202571.268849:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202571.268855:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4026 00000100:00080000:0.0:1788202571.268860:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202571.268867:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202571.268879:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202571.268893:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187723136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202571.268906:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788202571.268907:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187723008:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202571.268909:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202571.268914:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202571.268921:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4026 00000100:00080000:0.0:1788202571.268928:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202571.268932:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202571.268936:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202571.268946:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187723264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202571.268953:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202571.268956:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202571.268958:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202571.268961:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4026 00000100:00080000:0.0:1788202571.268963:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202571.268966:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202571.268970:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202571.268978:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187723392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202571.268984:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202571.268987:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202571.268989:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202571.268991:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4026 00000100:00080000:0.0:1788202571.268994:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202571.268998:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202571.269000:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202571.269008:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187723520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202571.269070:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187723136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202571.269136:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187723264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202571.269177:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187723392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202571.269218:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187723520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202571.972507:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202572.996546:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202574.020522:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788202574.340687:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788202574.340739:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788202574.340758:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202574.340806:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202574.340818:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202574.340827:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202574.340835:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:2.0:1788202574.340936:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202571/real 1788202574] req@ffff8e828793f480 x1875062187723136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202574.340976:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187723136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202574.340994:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202574.340999:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202574.341003:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187723136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202574.341011:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202571/real 1788202574] req@ffff8e828793dc00 x1875062187723264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202574.341023:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187723264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202574.341031:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202574.341034:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202574.341039:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187723264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202574.341046:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202571/real 1788202574] req@ffff8e828793c000 x1875062187723392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202574.341057:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187723392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202574.341065:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202574.341069:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202574.341073:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187723392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202574.341081:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202571/real 1788202574] req@ffff8e828793e300 x1875062187723520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202574.341094:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187723520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202574.341102:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202574.341526:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202574.341535:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187723520:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202574.341544:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202571/real 1788202574] req@ffff8e828793c700 x1875062187723008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202574.341563:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187723008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202596 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202574.341572:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202574.341576:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202574.341580:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187723008:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788202575.044484:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202576.068534:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202576.388448:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202576.388461:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202576.388465:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202576.388476:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4031 00000100:00080000:0.0:1788202576.388488:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202576.388496:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202576.388536:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187723648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202576.388564:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202576.388567:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202576.388569:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202576.388572:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4031 00000100:00080000:0.0:1788202576.388575:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202576.388577:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202576.388587:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202576.388597:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187723776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202576.388605:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202576.388608:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202576.388609:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202576.388612:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4031 00000100:00080000:0.0:1788202576.388613:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202576.388616:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202576.388619:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202576.388626:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187723648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202576.388641:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187723904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202576.388654:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202576.388658:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202576.388660:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202576.388665:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4031 00000100:00080000:0.0:1788202576.388670:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202576.388680:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202576.388688:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202576.388701:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187724032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202576.388711:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202576.388715:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202576.388717:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202576.388721:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4031 00000800:00000100:2.0:1788202576.388727:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202576.388729:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202576.388737:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202576.388742:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202576.388758:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187724160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202576.388793:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202576/real 1788202576] req@ffff8e828793f100 x1875062187723648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202576.388818:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187723648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202576.388831:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202576.388837:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202576.388841:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187723648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202576.388849:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187723776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202576.388881:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202576.388897:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202576/real 1788202576] req@ffff8e828793e680 x1875062187723776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202576.388911:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187723776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202576.388921:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202576.388925:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202576.388927:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187723776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202576.388958:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187723904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202576.388985:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202576.388998:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202576/real 1788202576] req@ffff8e828793f800 x1875062187723904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202576.389011:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187723904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202576.389020:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202576.389024:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202576.389027:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187723904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202576.389034:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187724032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202576.389055:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202576.389067:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202576/real 1788202576] req@ffff8e828793ce00 x1875062187724032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202576.389085:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187724032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202576.389093:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202576.389097:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202576.389100:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187724032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202576.389106:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187724160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202576.389126:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202576.389137:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202576/real 1788202576] req@ffff8e828793c700 x1875062187724160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202576.389155:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187724160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202576.389163:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202576.389195:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202576.389199:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187724160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202577.092514:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202578.116494:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202579.140505:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202580.164487:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202581.188473:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202581.508429:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202581.508439:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202581.508442:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202581.508451:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4036 00000100:00080000:2.0:1788202581.508459:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202581.508466:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202581.508498:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187724288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202581.508522:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202581.508525:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202581.508528:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202581.508532:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4036 00000100:00080000:2.0:1788202581.508536:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202581.508541:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202581.508550:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202581.508560:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187724288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202581.508563:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187724416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202581.508573:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202581.508576:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202581.508578:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202581.508581:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4036 00000100:00080000:2.0:1788202581.508583:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202581.508587:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202581.508591:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202581.508599:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187724544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202581.508607:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202581.508609:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202581.508611:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202581.508614:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4036 00000100:00080000:2.0:1788202581.508617:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788202581.508620:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788202581.508621:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202581.508625:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202581.508634:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187724672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202581.508642:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202581.508645:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202581.508646:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202581.508649:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4036 00000100:00080000:2.0:1788202581.508652:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202581.508655:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202581.508659:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202581.508668:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187724800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202581.508670:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202581/real 1788202581] req@ffff8e82894e4a80 x1875062187724288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202581.508697:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187724288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202581.508707:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202581.508712:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202581.508717:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187724288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202581.508740:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187724416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202581.508758:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202581.508767:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202581/real 1788202581] req@ffff8e82894e6300 x1875062187724416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202581.508775:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187724416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202581.508781:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202581.508783:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202581.508785:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187724416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202581.508789:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187724544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202581.508799:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202581.508805:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202581/real 1788202581] req@ffff8e82894e7800 x1875062187724544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202581.508812:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187724544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202581.508821:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202581.508823:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202581.508825:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187724544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202581.508828:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187724672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202581.508838:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202581.508844:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202581/real 1788202581] req@ffff8e82894e4e00 x1875062187724672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202581.508851:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187724672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202581.508856:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202581.508857:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202581.508859:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187724672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788202581.508862:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187724800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202581.508872:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202581.508877:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202581/real 1788202581] req@ffff8e82894e6680 x1875062187724800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202581.508884:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187724800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202581.508888:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202581.508890:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202581.508893:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187724800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202582.212473:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202583.236566:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202584.260482:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202585.284558:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202586.308464:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202586.628460:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202586.628473:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202586.628480:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202586.628494:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4041 00000100:00080000:2.0:1788202586.628508:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202586.628519:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202586.628577:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187724928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202586.628609:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202586.628616:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202586.628618:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202586.628623:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4041 00000100:00080000:2.0:1788202586.628627:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202586.628633:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202586.628645:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202586.628662:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187725056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202586.628668:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187724928:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202586.628676:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202586.628680:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202586.628685:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202586.628693:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4041 00000100:00080000:2.0:1788202586.628697:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202586.628701:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202586.628706:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202586.628719:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187725184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202586.628728:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202586.628732:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202586.628734:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202586.628738:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4041 00000100:00080000:2.0:1788202586.628742:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202586.628747:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202586.628753:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202586.628765:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187725312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202586.628776:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788202586.628782:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788202586.628783:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202586.628787:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202586.628792:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4041 00000100:00080000:2.0:1788202586.628796:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202586.628800:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202586.628805:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202586.628817:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187725440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202586.628856:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202586/real 1788202586] req@ffff8e82894e5880 x1875062187724928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202586.628890:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187724928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202586.628902:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202586.628907:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202586.628912:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187724928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202586.628943:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187725056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202586.628970:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202586.628984:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202586/real 1788202586] req@ffff8e82894e4000 x1875062187725056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202586.629029:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187725056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202586.629039:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202586.629043:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202586.629047:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187725056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202586.629054:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187725184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202586.629079:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202586.629093:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202586/real 1788202586] req@ffff8e82894e6a00 x1875062187725184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202586.629106:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187725184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202586.629115:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202586.629118:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202586.629122:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187725184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202586.629129:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187725312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202586.629150:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202586.629161:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202586/real 1788202586] req@ffff8e82894e7480 x1875062187725312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202586.629184:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187725312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202586.629192:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202586.629196:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202586.629200:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187725312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788202586.629206:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187725440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202586.629228:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202586.629240:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202586/real 1788202586] req@ffff8e82894e6680 x1875062187725440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202586.629262:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187725440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202586.629272:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202586.629276:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202586.629280:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187725440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202587.332498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202588.356484:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202589.380554:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202590.404458:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202591.428507:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202591.748456:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202591.748466:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202591.748470:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202591.748480:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4046 00000100:00080000:0.0:1788202591.748490:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202591.748497:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202591.748545:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187725568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202591.748574:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202591.748578:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202591.748580:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202591.748585:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4046 00000100:00080000:0.0:1788202591.748589:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202591.748594:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202591.748603:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202591.748609:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187725568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202591.748619:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187725696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202591.748633:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202591.748636:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202591.748639:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202591.748644:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4046 00000100:00080000:0.0:1788202591.748648:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202591.748653:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202591.748660:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788202591.748670:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202591.748675:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187725824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202591.748687:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202591.748691:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202591.748693:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202591.748696:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4046 00000100:00080000:0.0:1788202591.748700:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202591.748705:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202591.748711:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202591.748721:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202591/real 1788202591] req@ffff8e828793e300 x1875062187725568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202591.748725:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187725952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202591.748743:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788202591.748746:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187725568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202591.748752:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202591.748757:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202591.748761:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202591.748765:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4046 00000100:00080000:2.0:1788202591.748769:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:0.0:1788202591.748775:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788202591.748775:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187725568:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202591.748785:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202591.748790:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202591.748800:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187725696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202591.748806:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187726080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202591.748825:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202591.748837:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202591/real 1788202591] req@ffff8e828793c000 x1875062187725696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202591.748847:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187725696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202591.748852:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202591.748855:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202591.748857:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187725696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202591.748860:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187725824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202591.748872:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202591.748878:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202591/real 1788202591] req@ffff8e828793dc00 x1875062187725824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202591.748885:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187725824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202591.748890:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202591.748892:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202591.748894:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187725824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202591.748922:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187725952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202591.748939:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202591.748947:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202591/real 1788202591] req@ffff8e828793f480 x1875062187725952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202591.748960:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187725952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202591.748966:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202591.748968:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202591.748970:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187725952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202591.748981:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187726080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202591.748993:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202591.748999:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202591/real 1788202591] req@ffff8e828793ca80 x1875062187726080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202591.749006:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187726080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202591.749011:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202591.749013:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202591.749015:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187726080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202592.452572:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202593.476527:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202594.500547:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202595.524526:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202596.548511:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202596.868422:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202596.868434:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202596.868439:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202596.868451:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4051 00000100:00080000:0.0:1788202596.868463:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202596.868471:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202596.868517:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187726208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202596.868543:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202596.868546:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202596.868548:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202596.868551:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4051 00000100:00080000:0.0:1788202596.868554:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202596.868557:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202596.868565:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202596.868574:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187726336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202596.868582:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202596.868586:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202596.868588:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202596.868590:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187726208:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202596.868592:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4051 00000100:00080000:0.0:1788202596.868597:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202596.868602:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202596.868605:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202596.868613:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187726464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202596.868619:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202596.868622:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202596.868624:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202596.868626:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4051 00000100:00080000:0.0:1788202596.868629:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202596.868631:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202596.868634:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202596.868640:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187726592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202596.868646:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202596.868648:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202596.868649:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202596.868652:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4051 00000100:00080000:0.0:1788202596.868654:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202596.868657:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202596.868660:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202596.868668:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187726720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202596.868678:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202596.868736:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202596/real 1788202596] req@ffff8e828793d500 x1875062187726208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202596.868751:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187726208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202596.868766:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202596.868769:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202596.868774:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187726208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202596.868800:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187726336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202596.868839:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202596.868866:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202596/real 1788202596] req@ffff8e828793d880 x1875062187726336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202596.868876:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187726336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202596.868882:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202596.868885:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202596.868889:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187726336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202596.868896:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187726464:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202596.868919:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202596.868932:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202596/real 1788202596] req@ffff8e828793c380 x1875062187726464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202596.868944:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187726464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202596.868950:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202596.868952:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202596.868954:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187726464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202596.868959:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187726592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202596.868975:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202596.868983:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202596/real 1788202596] req@ffff8e828793fb80 x1875062187726592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202596.868990:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187726592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202596.868996:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202596.868998:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202596.869000:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187726592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202596.869003:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187726720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202596.869014:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202596.869024:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202596/real 1788202596] req@ffff8e828793ca80 x1875062187726720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202596.869034:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187726720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202596.869043:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202596.869046:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202596.869050:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187726720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202597.572546:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202598.596534:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202599.620492:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202600.644594:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202601.668527:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202601.988452:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202601.988465:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202601.988469:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202601.988480:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4057 00000100:00080000:0.0:1788202601.988493:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202601.988500:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202601.988549:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187726848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202601.988584:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202601.988588:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202601.988590:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202601.988596:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4057 00000100:00080000:0.0:1788202601.988600:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202601.988605:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202601.988616:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202601.988632:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187726976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202601.988642:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202601.988645:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788202601.988646:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187726848:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202601.988647:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202601.988651:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4057 00000100:00080000:0.0:1788202601.988654:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202601.988658:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202601.988663:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202601.988671:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187727104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202601.988681:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202601.988684:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202601.988686:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202601.988690:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4057 00000100:00080000:0.0:1788202601.988692:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202601.988696:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202601.988700:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202601.988711:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187727232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202601.988719:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202601.988739:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202601.988741:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788202601.988742:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202601.988745:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4057 00000100:00080000:0.0:1788202601.988748:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202601.988753:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202601.988757:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202601.988770:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187727360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202601.988801:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202601/real 1788202601] req@ffff8e828793f480 x1875062187726848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202601.988821:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187726848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202601.988833:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202601.988836:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202601.988839:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187726848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202601.988863:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187726976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202601.988882:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202601.988900:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202601/real 1788202601] req@ffff8e828793dc00 x1875062187726976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202601.988909:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187726976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202601.988914:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202601.988916:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202601.988919:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187726976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202601.988922:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187727104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202601.988934:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202601.988940:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202601/real 1788202601] req@ffff8e828793c000 x1875062187727104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202601.988948:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187727104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202601.988958:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202601.988960:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202601.988962:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187727104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202601.988965:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187727232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202601.988976:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202601.988989:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202601/real 1788202601] req@ffff8e828793e300 x1875062187727232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202601.988996:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187727232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202601.989001:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202601.989003:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202601.989005:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187727232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202601.989008:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187727360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202601.989020:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202601.989026:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202601/real 1788202601] req@ffff8e828793c700 x1875062187727360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202601.989033:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187727360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202601.989043:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202601.989046:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202601.989049:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187727360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202602.692521:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202603.716504:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202604.740549:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202605.764480:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202606.788540:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202607.108398:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202607.108409:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202607.108414:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202607.108425:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4062 00000100:00080000:0.0:1788202607.108438:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202607.108446:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202607.108498:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187727488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202607.108536:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202607.108539:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202607.108541:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202607.108545:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4062 00000100:00080000:0.0:1788202607.108549:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202607.108554:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202607.108566:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202607.108571:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187727488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202607.108586:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187727616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202607.108601:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202607.108606:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202607.108610:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202607.108617:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4062 00000100:00080000:0.0:1788202607.108621:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202607.108625:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202607.108645:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202607.108660:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187727744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202607.108664:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202607.108675:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202607.108680:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202607.108682:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202607.108687:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4062 00000100:00080000:0.0:1788202607.108692:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202607.108696:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202607.108703:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202607.108709:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202607/real 1788202607] req@ffff8e828793ce00 x1875062187727488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202607.108719:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187727872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202607.108730:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788202607.108733:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187727488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202607.108735:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202607.108739:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202607.108749:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4062 00000100:00080000:0.0:1788202607.108755:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202607.108757:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202607.108762:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:2.0:1788202607.108767:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202607.108774:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187727488:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788202607.108780:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202607.108798:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187728000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202607.108808:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187727616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202607.108837:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202607.108854:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202607/real 1788202607] req@ffff8e828793f800 x1875062187727616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202607.108880:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187727616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202607.108890:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202607.108894:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202607.108896:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187727616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202607.108902:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187727744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202607.108923:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202607.108938:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202607/real 1788202607] req@ffff8e828793e680 x1875062187727744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202607.108952:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187727744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202607.108963:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202607.108967:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202607.108972:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187727744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202607.108978:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187727872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202607.109000:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202607.109011:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202607/real 1788202607] req@ffff8e828793f100 x1875062187727872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202607.109020:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187727872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202607.109027:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202607.109030:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202607.109034:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187727872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202607.109058:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187728000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202607.109070:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202607.109077:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202607/real 1788202607] req@ffff8e828793c700 x1875062187728000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202632 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202607.109084:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187728000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202632 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202607.109090:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202607.109092:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202607.109094:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187728000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202607.812473:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202608.836483:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202609.860523:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202610.884543:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202611.908524:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202612.228433:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202612.228443:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202612.228454:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202612.228466:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4067 00000100:00080000:0.0:1788202612.228478:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202612.228486:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202612.228531:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187728128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202612.228569:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202612.228573:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202612.228574:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202612.228578:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4067 00000100:00080000:0.0:1788202612.228581:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202612.228584:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202612.228594:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202612.228605:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187728256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202612.228614:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202612.228617:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202612.228620:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202612.228623:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187728128:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202612.228624:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4067 00000100:00080000:0.0:1788202612.228630:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202612.228635:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202612.228638:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202612.228646:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187728384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202612.228653:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202612.228655:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202612.228657:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202612.228659:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4067 00000100:00080000:0.0:1788202612.228662:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202612.228666:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202612.228670:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202612.228677:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187728512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202612.228684:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202612.228687:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202612.228688:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202612.228690:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4067 00000100:00080000:0.0:1788202612.228692:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202612.228694:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202612.228697:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202612.228714:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187728640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202612.228735:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202612.228791:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202612/real 1788202612] req@ffff8e828793e300 x1875062187728128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202612.228807:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187728128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202612.228816:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202612.228819:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202612.228823:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187728128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202612.228845:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187728256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202612.228864:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202612.228881:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202612/real 1788202612] req@ffff8e828793c000 x1875062187728256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202612.228889:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187728256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202612.228895:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202612.228897:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202612.228899:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187728256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202612.228904:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187728384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202612.228917:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202612.228923:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202612/real 1788202612] req@ffff8e828793dc00 x1875062187728384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202612.228931:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187728384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202612.228936:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202612.228938:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202612.228940:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187728384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202612.228944:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187728512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202612.228956:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202612.228962:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202612/real 1788202612] req@ffff8e828793f480 x1875062187728512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202612.228969:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187728512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202612.228974:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202612.228976:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202612.228977:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187728512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202612.228980:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187728640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202612.228990:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202612.228995:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202612/real 1788202612] req@ffff8e828793ca80 x1875062187728640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202637 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202612.229003:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187728640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202637 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202612.229008:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202612.229010:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202612.229012:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187728640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202612.932505:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202613.956502:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202614.980548:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202616.004548:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202617.028503:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202617.348468:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202617.348481:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202617.348486:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202617.348498:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4072 00000100:00080000:3.0:1788202617.348511:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202617.348518:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202617.348565:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187728768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202617.348593:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202617.348596:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202617.348598:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202617.348602:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4072 00000100:00080000:3.0:1788202617.348605:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202617.348608:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202617.348616:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202617.348625:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187728896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202617.348633:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202617.348641:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202617.348643:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202617.348646:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4072 00000100:00080000:3.0:1788202617.348649:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202617.348652:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202617.348655:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202617.348659:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187728768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202617.348668:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187729024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202617.348687:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202617.348691:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202617.348694:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202617.348699:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4072 00000100:00080000:3.0:1788202617.348718:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202617.348724:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202617.348729:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788202617.348734:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202617.348746:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187729152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202617.348757:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202617.348762:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202617.348764:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202617.348768:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4072 00000100:00080000:3.0:1788202617.348773:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202617.348779:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202617.348783:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202617/real 1788202617] req@ffff8e8289538a80 x1875062187728768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788202617.348787:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202617.348805:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187729280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202617.348812:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187728768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202617.348825:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202617.348828:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202617.348832:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187728768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202617.348839:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187728896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202617.348856:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202617.348865:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202617/real 1788202617] req@ffff8e828953a680 x1875062187728896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202617.348873:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187728896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202617.348884:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202617.348886:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202617.348889:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187728896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202617.348919:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187729024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202617.348932:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202617.348939:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202617/real 1788202617] req@ffff8e8289539880 x1875062187729024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202617.348947:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187729024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202617.348955:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202617.348958:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202617.348962:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187729024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202617.348968:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187729152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202617.348990:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202617.349003:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202617/real 1788202617] req@ffff8e8289539500 x1875062187729152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202617.349016:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187729152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202617.349022:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202617.349024:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202617.349027:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187729152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202617.349032:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187729280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202617.349047:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202617.349054:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202617/real 1788202617] req@ffff8e828953b100 x1875062187729280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202617.349062:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187729280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202617.349067:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202617.349069:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202617.349071:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187729280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202618.052520:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202619.076520:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202620.100501:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202621.124485:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202622.148526:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202622.468426:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202622.468440:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202622.468445:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202622.468459:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4077 00000100:00080000:2.0:1788202622.468473:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202622.468482:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202622.468533:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187729408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202622.468560:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202622.468563:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202622.468567:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202622.468571:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4077 00000100:00080000:2.0:1788202622.468574:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202622.468577:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202622.468588:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202622.468603:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187729536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202622.468613:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788202622.468614:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187729408:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202622.468616:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202622.468620:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202622.468628:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4077 00000100:00080000:2.0:1788202622.468631:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202622.468633:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202622.468636:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202622.468644:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187729664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202622.468650:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202622.468652:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202622.468654:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202622.468656:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4077 00000100:00080000:2.0:1788202622.468658:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202622.468661:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202622.468665:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202622.468672:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187729792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202622.468678:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202622.468681:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202622.468683:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202622.468685:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4077 00000100:00080000:2.0:1788202622.468688:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202622.468690:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202622.468693:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202622.468699:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187729920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202622.468712:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202622.468763:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202622/real 1788202622] req@ffff8e82894e4e00 x1875062187729408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202622.468779:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187729408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202622.468789:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202622.468792:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202622.468796:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187729408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202622.468819:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187729536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202622.468837:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202622.468846:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202622/real 1788202622] req@ffff8e82894e7800 x1875062187729536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202622.468854:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187729536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202622.468860:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202622.468863:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202622.468865:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187729536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202622.468869:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187729664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202622.468879:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202622.468886:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202622/real 1788202622] req@ffff8e82894e6300 x1875062187729664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202622.468893:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187729664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202622.468899:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202622.468900:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202622.468902:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187729664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202622.468906:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187729792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202622.468917:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202622.468923:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202622/real 1788202622] req@ffff8e82894e4a80 x1875062187729792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202622.468931:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187729792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202622.468937:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202622.468938:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202622.468940:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187729792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788202622.468944:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187729920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202622.468954:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202622.468960:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202622/real 1788202622] req@ffff8e82894e7100 x1875062187729920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202622.468967:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187729920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202622.468972:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202622.468974:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202622.468975:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187729920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202623.172489:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202624.196546:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202625.220486:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202626.244525:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202627.268445:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202627.588477:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202627.588492:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202627.588499:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202627.588513:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4082 00000100:00080000:0.0:1788202627.588528:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202627.588538:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202627.588588:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187730048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202627.588622:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202627.588627:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202627.588629:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202627.588635:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4082 00000100:00080000:0.0:1788202627.588640:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202627.588645:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202627.588658:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202627.588668:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187730048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202627.588676:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187730176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202627.588690:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202627.588695:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202627.588697:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202627.588701:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4082 00000100:00080000:0.0:1788202627.588706:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202627.588711:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202627.588717:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202627.588729:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187730304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202627.588740:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202627.588746:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1788202627.588747:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202627.588751:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202627.588757:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4082 00000100:00080000:0.0:1788202627.588763:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202627.588769:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202627.588784:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202627.588795:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202627/real 1788202627] req@ffff8e828793fb80 x1875062187730048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202627.588807:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187730432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202627.588815:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187730048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202627.588827:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202627.588831:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202627.588833:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202627.588837:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:0.0:1788202627.588838:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202627.588843:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187730048:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202627.588849:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4082 00000100:00080000:0.0:1788202627.588856:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202627.588861:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202627.588867:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202627.588872:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187730176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202627.588886:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187730560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202627.588896:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202627.588910:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202627/real 1788202627] req@ffff8e828793c380 x1875062187730176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202627.588920:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187730176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202627.588928:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202627.588931:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202627.588934:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187730176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202627.588938:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187730304:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202627.588950:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202627.588957:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202627/real 1788202627] req@ffff8e828793d880 x1875062187730304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202627.588972:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187730304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202627.588978:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202627.588981:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202627.588983:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187730304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202627.588986:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187730432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202627.588999:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202627.589006:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202627/real 1788202627] req@ffff8e828793d500 x1875062187730432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202627.589013:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187730432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202627.589018:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202627.589020:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202627.589023:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187730432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202627.589033:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187730560:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202627.589044:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202627.589050:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202627/real 1788202627] req@ffff8e828793fb80 x1875062187730560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202627.589058:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187730560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202627.589063:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202627.589064:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202627.589066:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187730560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202628.292510:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202629.316490:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202630.340484:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202631.364537:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202632.388528:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202632.708445:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202632.708459:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202632.708465:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202632.708475:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4087 00000100:00080000:3.0:1788202632.708489:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202632.708498:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202632.708549:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187730688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202632.708573:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202632.708576:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202632.708577:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202632.708581:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4087 00000100:00080000:3.0:1788202632.708584:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202632.708588:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202632.708598:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202632.708608:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187730688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202632.708613:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187730816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202632.708623:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202632.708625:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202632.708626:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202632.708628:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4087 00000100:00080000:3.0:1788202632.708630:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202632.708632:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202632.708635:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202632.708644:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187730944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202632.708649:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202632.708653:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202632.708654:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202632.708657:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4087 00000100:00080000:3.0:1788202632.708660:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202632.708662:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202632.708665:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202632.708672:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187731072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202632.708678:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202632.708680:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202632.708681:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202632.708683:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4087 00000100:00080000:3.0:1788202632.708686:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202632.708689:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202632.708691:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202632.708698:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187731200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202632.708707:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202632.708784:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202632/real 1788202632] req@ffff8e828953bb80 x1875062187730688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202632.708802:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187730688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202632.708810:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202632.708813:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202632.708816:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187730688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202632.708841:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187730816:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202632.708867:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202632.708883:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202632/real 1788202632] req@ffff8e8289538380 x1875062187730816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202632.708896:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187730816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202632.708904:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202632.708908:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202632.708911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187730816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202632.708917:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187730944:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202632.708938:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202632.708950:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202632/real 1788202632] req@ffff8e8289539f80 x1875062187730944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202632.708961:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187730944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202632.708969:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202632.708972:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202632.708975:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187730944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202632.708982:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187731072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202632.709003:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202632.709015:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202632/real 1788202632] req@ffff8e828953b100 x1875062187731072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202632.709027:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187731072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202632.709036:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202632.709040:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202632.709043:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187731072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202632.709050:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187731200:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202632.709072:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202632.709084:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202632/real 1788202632] req@ffff8e8289539500 x1875062187731200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202632.709096:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187731200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202632.709105:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202632.709108:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202632.709121:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187731200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202633.412543:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202634.436574:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202635.460516:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202636.484516:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202637.508529:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202637.828453:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202637.828468:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202637.828473:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202637.828485:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4092 00000100:00080000:3.0:1788202637.828497:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202637.828504:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202637.828547:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187731328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202637.828574:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202637.828577:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202637.828579:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202637.828582:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4092 00000100:00080000:3.0:1788202637.828585:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202637.828595:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202637.828607:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202637.828613:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187731328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202637.828623:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187731456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202637.828637:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202637.828641:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202637.828643:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202637.828647:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4092 00000100:00080000:3.0:1788202637.828651:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202637.828656:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202637.828662:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202637.828678:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187731584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202637.828690:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202637.828694:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202637.828696:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202637.828700:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4092 00000100:00080000:3.0:1788202637.828706:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202637.828711:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202637.828717:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202637.828731:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187731712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202637.828741:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202637.828745:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202637.828748:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788202637.828750:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187731456:192.168.201.121@tcp:38:kworker.0 00000100:00080000:3.0:1788202637.828755:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4092 00000100:00080000:3.0:1788202637.828761:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202637.828766:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202637.828773:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202637.828788:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187731840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202637.828791:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187731584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202637.828825:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187731712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788202637.828853:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187731840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202638.532482:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202639.556541:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202640.580520:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788202640.900712:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:1.0:1788202640.900775:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788202640.900798:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788202640.900851:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788202640.900863:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788202640.900876:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788202640.900888:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:0.0:1788202640.900919:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202637/real 1788202640] req@ffff8e828953b100 x1875062187731840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202640.900964:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187731840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202640.900980:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202640.900987:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202640.900992:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187731840:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788202640.901002:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202637/real 1788202640] req@ffff8e828953a680 x1875062187731456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202640.901020:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187731456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202640.901029:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202640.901033:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202640.901036:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187731456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202640.901057:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202637/real 1788202640] req@ffff8e8289538a80 x1875062187731584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202640.901072:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187731584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202640.901080:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202640.901085:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202640.901088:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187731584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202640.901095:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202637/real 1788202640] req@ffff8e8289539500 x1875062187731712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202640.901108:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187731712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202640.901116:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202640.901120:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202640.901124:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187731712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788202640.901131:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202637/real 1788202640] req@ffff8e8289539880 x1875062187731328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202640.901143:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187731328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202640.901151:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202640.901154:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202640.901157:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187731328:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788202641.604538:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202642.628448:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202642.948378:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202642.948386:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202642.948390:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202642.948399:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4098 00000100:00080000:3.0:1788202642.948408:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202642.948416:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202642.948446:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187731968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202642.948469:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202642.948475:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202642.948478:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202642.948483:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4098 00000100:00080000:3.0:1788202642.948488:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788202642.948501:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187731968:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202642.948507:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202642.948519:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202642.948538:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187732096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202642.948551:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202642.948554:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202642.948557:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202642.948562:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4098 00000100:00080000:3.0:1788202642.948568:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788202642.948573:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788202642.948577:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202642.948585:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202642.948602:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187732224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202642.948613:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202642.948617:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202642.948620:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788202642.948626:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202642/real 1788202642] req@ffff8e8289539f80 x1875062187731968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202642.948627:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4098 00000100:00080000:3.0:1788202642.948635:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202642.948645:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202642.948653:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202642.948654:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187731968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202642.948669:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788202642.948673:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187732352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202642.948676:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202642.948686:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187731968:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202642.948693:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202642.948699:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202642.948701:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202642.948708:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4098 00000100:00080000:3.0:1788202642.948712:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202642.948717:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202642.948721:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187732096:192.168.201.121@tcp:38:kworker.0 00000008:00100000:3.0:1788202642.948725:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202642.948743:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187732480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202642.948757:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202642.948792:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202642/real 1788202642] req@ffff8e8289538380 x1875062187732096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202642.948807:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187732096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202642.948815:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202642.948819:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202642.948830:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187732096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202642.948841:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187732224:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202642.948862:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202642.948872:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202642/real 1788202642] req@ffff8e828953bb80 x1875062187732224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202642.948884:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187732224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202642.948893:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202642.948896:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202642.948900:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187732224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202642.948905:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187732352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202642.948920:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202642.948930:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202642/real 1788202642] req@ffff8e8289539880 x1875062187732352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202642.948942:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187732352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202642.948951:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202642.948954:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202642.948958:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187732352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788202642.948973:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187732480:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202642.948996:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202642.949007:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202642/real 1788202642] req@ffff8e8289539500 x1875062187732480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202642.949037:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187732480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202642.949045:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202642.949049:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202642.949052:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187732480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202643.652465:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202644.677436:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202645.700450:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202646.724467:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202647.748429:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202648.068425:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202648.068439:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202648.068445:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202648.068459:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4103 00000100:00080000:0.0:1788202648.068473:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202648.068483:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202648.068547:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187732608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202648.068583:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202648.068587:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202648.068590:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202648.068596:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4103 00000100:00080000:0.0:1788202648.068601:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202648.068607:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202648.068616:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187732608:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788202648.068619:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202648.068638:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187732736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202648.068650:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202648.068655:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202648.068657:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202648.068660:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4103 00000100:00080000:0.0:1788202648.068665:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202648.068672:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202648.068678:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202648.068692:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187732864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202648.068694:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202648.068708:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202648.068712:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202648.068715:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202648.068720:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4103 00000100:00080000:0.0:1788202648.068725:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202648.068731:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202648.068737:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202648.068746:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202648/real 1788202648] req@ffff8e828793ca80 x1875062187732608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202648.068752:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187732992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202648.068773:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202648.068777:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202648.068781:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202648.068782:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187732608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202648.068791:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4103 00000100:00080000:2.0:1788202648.068804:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202648.068808:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202648.068816:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:2.0:1788202648.068820:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000008:00100000:0.0:1788202648.068822:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202648.068826:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187732608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202648.068838:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187733120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202648.068859:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187732736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202648.068891:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202648.068916:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202648/real 1788202648] req@ffff8e828793f480 x1875062187732736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202648.068941:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187732736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202648.068948:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202648.068950:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202648.068953:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187732736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202648.068957:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187732864:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202648.068975:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202648.068983:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202648/real 1788202648] req@ffff8e828793dc00 x1875062187732864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202648.068991:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187732864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202648.068997:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202648.068999:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202648.069001:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187732864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202648.069004:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187732992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202648.069016:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202648.069023:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202648/real 1788202648] req@ffff8e828793c000 x1875062187732992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202648.069031:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187732992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202648.069036:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202648.069038:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202648.069040:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187732992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202648.069043:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187733120:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202648.069054:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202648.069060:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202648/real 1788202648] req@ffff8e828793e300 x1875062187733120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202673 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202648.069076:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187733120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202673 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202648.069082:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202648.069084:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202648.069086:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187733120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202648.772472:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202649.796496:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202650.820500:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202651.844515:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202652.868498:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202653.188461:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202653.188472:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202653.188477:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202653.188488:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4108 00000100:00080000:1.0:1788202653.188500:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202653.188509:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202653.188554:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187733248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202653.188580:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202653.188583:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202653.188584:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202653.188587:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4108 00000100:00080000:1.0:1788202653.188590:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202653.188600:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202653.188612:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202653.188622:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187733376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202653.188630:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202653.188633:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202653.188635:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202653.188638:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4108 00000100:00080000:1.0:1788202653.188640:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202653.188643:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202653.188646:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202653.188654:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187733504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202653.188661:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187733248:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202653.188663:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202653.188672:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202653.188674:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202653.188677:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4108 00000100:00080000:1.0:1788202653.188680:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202653.188683:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202653.188687:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202653.188696:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187733632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202653.188703:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202653.188705:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202653.188707:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202653.188709:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4108 00000100:00080000:1.0:1788202653.188711:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202653.188714:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202653.188717:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202653.188724:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187733760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202653.188755:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202653.188809:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202653/real 1788202653] req@ffff8e828d722680 x1875062187733248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202653.188856:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187733248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202653.188872:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202653.188875:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202653.188878:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187733248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202653.188884:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187733376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202653.188903:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202653.188913:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202653/real 1788202653] req@ffff8e828d723800 x1875062187733376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202653.188921:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187733376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202653.188927:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202653.188929:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202653.188932:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187733376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202653.188955:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187733504:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202653.188973:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202653.188985:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202653/real 1788202653] req@ffff8e828d721500 x1875062187733504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202653.188998:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187733504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202653.189007:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202653.189010:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202653.189014:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187733504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202653.189021:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187733632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202653.189041:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202653.189049:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202653/real 1788202653] req@ffff8e828d720000 x1875062187733632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202653.189057:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187733632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202653.189063:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202653.189065:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202653.189067:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187733632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202653.189071:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187733760:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202653.189081:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202653.189087:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202653/real 1788202653] req@ffff8e828d720e00 x1875062187733760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202653.189094:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187733760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202653.189099:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202653.189100:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202653.189102:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187733760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202653.892506:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202654.916502:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202655.940580:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202656.964481:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202657.988509:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202658.308512:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202658.308522:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202658.308527:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202658.308539:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4113 00000100:00080000:2.0:1788202658.308551:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202658.308559:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202658.308604:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187733888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202658.308646:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202658.308651:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202658.308653:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202658.308658:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4113 00000100:00080000:2.0:1788202658.308663:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202658.308668:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202658.308679:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202658.308695:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187734016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788202658.308706:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187733888:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202658.308713:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202658.308719:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202658.308721:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202658.308726:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4113 00000100:00080000:2.0:1788202658.308730:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202658.308735:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202658.308757:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202658.308773:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187734144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202658.308809:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202658.308814:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202658.308816:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202658.308823:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4113 00000100:00080000:2.0:1788202658.308830:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202658.308838:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:0.0:1788202658.308843:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:2.0:1788202658.308850:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202658.308871:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187734272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202658.308884:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202658.308890:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202658.308893:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202658.308900:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4113 00000100:00080000:2.0:1788202658.308906:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788202658.308925:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202658/real 1788202658] req@ffff8e82894e6d80 x1875062187733888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202658.308930:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202658.308940:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202658.308954:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187733888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202658.308958:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187734400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202658.308987:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202658.308994:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202658.309001:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187733888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202658.309034:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187734016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202658.309056:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202658.309075:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202658/real 1788202658] req@ffff8e82894e5f80 x1875062187734016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202658.309087:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187734016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202658.309094:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202658.309097:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202658.309100:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187734016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202658.309104:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187734144:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202658.309119:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202658.309126:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202658/real 1788202658] req@ffff8e82894e4380 x1875062187734144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202658.309135:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187734144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202658.309140:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202658.309143:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202658.309145:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187734144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202658.309149:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187734272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202658.309161:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202658.309168:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202658/real 1788202658] req@ffff8e82894e5c00 x1875062187734272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202658.309177:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187734272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202658.309183:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202658.309185:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202658.309187:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187734272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788202658.309191:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187734400:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202658.309202:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202658.309208:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202658/real 1788202658] req@ffff8e82894e7100 x1875062187734400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202658.309217:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187734400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202658.309222:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202658.309224:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202658.309226:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187734400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202659.012594:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202660.036479:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202661.060533:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202662.084468:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202663.108481:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202663.428415:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202663.428423:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202663.428427:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202663.428435:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4118 00000100:00080000:1.0:1788202663.428444:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202663.428451:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202663.428483:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187734528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202663.428516:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202663.428520:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202663.428527:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202663.428533:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4118 00000100:00080000:1.0:1788202663.428537:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788202663.428543:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187734528:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202663.428545:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202663.428558:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202663.428576:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187734656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202663.428589:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202663.428591:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202663.428593:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202663.428596:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4118 00000100:00080000:1.0:1788202663.428600:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202663.428603:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:2.0:1788202663.428607:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:1.0:1788202663.428609:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202663.428621:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187734784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202663.428629:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202663.428632:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202663.428633:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202663.428636:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4118 00000100:00080000:1.0:1788202663.428639:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202663.428642:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202663.428646:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202663.428647:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202663/real 1788202663] req@ffff8e828d720380 x1875062187734528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202663.428672:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187734912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202663.428701:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202663.428705:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202663.428707:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202663.428711:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187734528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202663.428712:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4118 00000100:00080000:1.0:1788202663.428718:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202663.428724:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:2.0:1788202663.428728:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1788202663.428729:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788202663.428735:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202663.428739:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187734528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202663.428743:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187735040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202663.428764:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187734656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202663.428782:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202663.428791:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202663/real 1788202663] req@ffff8e828d722300 x1875062187734656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202663.428799:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187734656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202663.428804:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202663.428806:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202663.428808:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187734656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202663.428811:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187734784:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202663.428821:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202663.428827:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202663/real 1788202663] req@ffff8e828d722d80 x1875062187734784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202663.428834:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187734784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202663.428843:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202663.428845:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202663.428847:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187734784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202663.428851:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187734912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202663.428860:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202663.428866:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202663/real 1788202663] req@ffff8e828d721180 x1875062187734912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202663.428872:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187734912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202663.428877:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202663.428878:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202663.428880:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187734912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202663.428883:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187735040:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202663.428892:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202663.428898:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202663/real 1788202663] req@ffff8e828d720a80 x1875062187735040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202663.428904:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187735040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202663.428909:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202663.428910:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202663.428913:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187735040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202664.132505:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202665.156511:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202666.180548:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202667.204474:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202668.228486:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202668.548415:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202668.548425:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202668.548431:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202668.548442:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4123 00000100:00080000:1.0:1788202668.548454:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202668.548463:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202668.548502:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187735168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202668.548528:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202668.548532:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202668.548534:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202668.548540:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4123 00000100:00080000:1.0:1788202668.548544:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202668.548549:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202668.548558:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187735168:192.168.201.121@tcp:250:kworker.0 00000008:00100000:1.0:1788202668.548559:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202668.548591:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187735296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202668.548603:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202668.548607:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202668.548610:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202668.548613:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4123 00000100:00080000:1.0:1788202668.548620:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202668.548624:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:2.0:1788202668.548630:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:1.0:1788202668.548631:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202668.548646:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187735424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202668.548662:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202668.548666:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202668.548668:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202668.548675:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202668/real 1788202668] req@ffff8e828d720a80 x1875062187735168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202668.548676:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4123 00000100:00080000:1.0:1788202668.548683:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202668.548692:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202668.548697:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187735168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788202668.548699:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788202668.548710:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202668.548715:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202668.548720:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187735552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202668.548725:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187735168:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202668.548740:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202668.548744:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202668.548746:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202668.548751:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4123 00000100:00080000:1.0:1788202668.548755:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788202668.548760:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187735296:192.168.201.121@tcp:38:kworker.0 00000100:00080000:1.0:1788202668.548761:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202668.548770:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202668.548784:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187735680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202668.548794:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202668.548809:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202668/real 1788202668] req@ffff8e828d721180 x1875062187735296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202668.548818:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187735296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202668.548825:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202668.548827:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202668.548829:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187735296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202668.548833:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187735424:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202668.548844:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202668.548851:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202668/real 1788202668] req@ffff8e828d722d80 x1875062187735424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202668.548859:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187735424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202668.548863:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202668.548865:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202668.548867:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187735424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202668.548871:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187735552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202668.548882:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202668.548889:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202668/real 1788202668] req@ffff8e828d722300 x1875062187735552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202668.548901:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187735552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202668.548910:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202668.548913:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202668.548917:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187735552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202668.548936:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187735680:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202668.548959:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202668.548971:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202668/real 1788202668] req@ffff8e828d720380 x1875062187735680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202668.548984:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187735680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202668.548993:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202668.548997:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202668.549001:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187735680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202669.252520:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202670.276501:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202671.300534:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202672.324521:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202673.348530:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202673.668456:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202673.668465:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202673.668469:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202673.668495:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4128 00000100:00080000:0.0:1788202673.668504:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202673.668511:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202673.668549:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187735808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202673.668576:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202673.668580:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202673.668582:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202673.668587:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4128 00000100:00080000:0.0:1788202673.668590:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202673.668600:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202673.668609:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202673.668624:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187735936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202673.668636:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202673.668642:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202673.668644:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202673.668647:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187735808:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202673.668649:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4128 00000100:00080000:0.0:1788202673.668657:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202673.668663:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202673.668668:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202673.668679:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187736064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202673.668687:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202673.668690:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202673.668692:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202673.668696:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4128 00000100:00080000:0.0:1788202673.668700:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202673.668704:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202673.668710:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788202673.668718:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202673.668725:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187736192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202673.668737:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202673.668741:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202673.668743:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202673.668747:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4128 00000100:00080000:0.0:1788202673.668752:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202673.668757:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202673.668762:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202673.668766:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202673/real 1788202673] req@ffff8e828793c700 x1875062187735808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202673.668778:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187736320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202673.668791:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187735808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202673.668800:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202673.668804:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202673.668807:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187735808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202673.668827:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187735936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202673.668846:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202673.668855:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202673/real 1788202673] req@ffff8e828793f100 x1875062187735936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202673.668863:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187735936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202673.668869:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202673.668871:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202673.668873:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187735936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202673.668878:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187736064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202673.668896:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202673.668908:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202673/real 1788202673] req@ffff8e828793e680 x1875062187736064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202673.668920:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187736064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202673.668929:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202673.668933:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202673.668936:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187736064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202673.668942:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187736192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202673.668959:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202673.668966:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202673/real 1788202673] req@ffff8e828793f800 x1875062187736192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202673.668974:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187736192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202673.668979:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202673.668981:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202673.668983:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187736192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202673.668987:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187736320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202673.668997:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202673.669002:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202673/real 1788202673] req@ffff8e828793ce00 x1875062187736320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202673.669010:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187736320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202673.669015:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202673.669016:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202673.669019:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187736320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202674.372587:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202675.396564:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202676.420545:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202677.444531:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202678.468551:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202678.788407:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202678.788424:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202678.788431:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202678.788446:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4133 00000100:00080000:0.0:1788202678.788461:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202678.788471:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202678.788518:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187736448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202678.788556:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202678.788560:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202678.788563:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202678.788569:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4133 00000100:00080000:0.0:1788202678.788575:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202678.788590:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202678.788606:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187736448:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788202678.788608:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202678.788638:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187736576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202678.788654:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202678.788658:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202678.788660:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202678.788665:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4133 00000100:00080000:0.0:1788202678.788685:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202678.788691:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:1.0:1788202678.788697:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788202678.788698:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202678.788716:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187736704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202678.788730:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202678.788735:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202678.788737:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202678.788742:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4133 00000100:00080000:0.0:1788202678.788747:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202678.788752:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202678/real 1788202678] req@ffff8e828793ce00 x1875062187736448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202678.788754:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202678.788764:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202678.788777:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187736448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202678.788786:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187736832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202678.788789:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202678.788797:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:0.0:1788202678.788805:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788202678.788806:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187736448:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202678.788813:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202678.788817:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202678.788821:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4133 00000100:00080000:0.0:1788202678.788825:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202678.788830:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202678.788836:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202678.788843:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187736576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202678.788851:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187736960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202678.788877:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202678.788895:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202678/real 1788202678] req@ffff8e828793f800 x1875062187736576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202678.788910:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187736576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202678.788920:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202678.788924:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202678.788926:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187736576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202678.788931:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187736704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202678.788956:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202678.788965:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202678/real 1788202678] req@ffff8e828793e680 x1875062187736704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202678.788978:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187736704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202678.788984:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202678.788986:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202678.788988:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187736704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202678.788992:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187736832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202678.789005:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202678.789013:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202678/real 1788202678] req@ffff8e828793f100 x1875062187736832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202678.789020:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187736832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202678.789025:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202678.789027:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202678.789029:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187736832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202678.789042:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187736960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202678.789055:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202678.789061:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202678/real 1788202678] req@ffff8e828793c700 x1875062187736960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202678.789077:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187736960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202678.789088:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202678.789090:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202678.789093:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187736960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202679.492480:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202680.516497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202681.540473:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202682.564513:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202683.588473:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202683.908420:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202683.908428:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202683.908432:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202683.908441:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4138 00000100:00080000:0.0:1788202683.908449:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202683.908457:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202683.908491:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187737088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202683.908511:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202683.908514:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202683.908515:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202683.908518:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4138 00000100:00080000:0.0:1788202683.908520:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202683.908523:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202683.908536:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202683.908548:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187737216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202683.908556:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202683.908560:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202683.908562:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788202683.908562:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187737088:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202683.908570:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4138 00000100:00080000:0.0:1788202683.908573:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202683.908575:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202683.908579:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202683.908589:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187737344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202683.908600:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202683.908602:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202683.908604:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202683.908606:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4138 00000100:00080000:0.0:1788202683.908608:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202683.908610:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202683.908613:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202683.908620:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187737472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202683.908626:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202683.908628:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202683.908630:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202683.908632:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4138 00000100:00080000:0.0:1788202683.908634:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202683.908636:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202683.908639:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788202683.908644:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202683.908649:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187737600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202683.908688:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202683/real 1788202683] req@ffff8e828793e300 x1875062187737088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202683.908702:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187737088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202683.908709:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202683.908712:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202683.908715:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187737088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202683.908733:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187737216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202683.908750:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202683.908758:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202683/real 1788202683] req@ffff8e828793c000 x1875062187737216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202683.908766:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187737216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202683.908772:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202683.908774:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202683.908777:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187737216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202683.908780:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187737344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202683.908791:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202683.908797:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202683/real 1788202683] req@ffff8e828793dc00 x1875062187737344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202683.908803:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187737344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202683.908808:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202683.908810:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202683.908812:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187737344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202683.908815:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187737472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202683.908825:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202683.908831:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202683/real 1788202683] req@ffff8e828793f480 x1875062187737472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202683.908837:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187737472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202683.908842:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202683.908844:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202683.908846:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187737472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202683.908850:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187737600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202683.908860:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202683.908865:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202683/real 1788202683] req@ffff8e828793ca80 x1875062187737600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202683.908873:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187737600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202683.908877:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202683.908879:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202683.908881:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187737600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202684.612494:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202685.636464:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202686.660533:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202687.684531:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202688.708543:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202689.028446:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202689.028461:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202689.028467:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202689.028490:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4144 00000100:00080000:1.0:1788202689.028504:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202689.028514:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202689.028563:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187737728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202689.028597:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202689.028602:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202689.028604:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202689.028610:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4144 00000100:00080000:1.0:1788202689.028615:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202689.028621:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202689.028633:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202689.028641:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187737728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202689.028651:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187737856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202689.028663:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202689.028668:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202689.028670:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202689.028674:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4144 00000100:00080000:1.0:1788202689.028678:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202689.028683:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202689.028688:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202689.028700:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187737984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202689.028711:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202689.028715:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202689.028717:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788202689.028722:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788202689.028725:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4144 00000100:00080000:1.0:1788202689.028731:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202689.028736:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202689.028742:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202689.028755:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187738112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202689.028765:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202689.028770:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202689.028771:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202689.028775:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4144 00000100:00080000:1.0:1788202689.028780:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202689.028785:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202689.028789:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202689/real 1788202689] req@ffff8e828d720e00 x1875062187737728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788202689.028792:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202689.028810:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187738240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202689.028818:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187737728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202689.028836:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202689.028842:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202689.028855:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187737728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202689.028883:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187737856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202689.028903:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202689.028913:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202689/real 1788202689] req@ffff8e828d720000 x1875062187737856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202689.028921:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187737856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202689.028932:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202689.028934:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202689.028937:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187737856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202689.028940:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187737984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202689.028961:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202689.028968:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202689/real 1788202689] req@ffff8e828d721500 x1875062187737984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202689.028976:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187737984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202689.028982:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202689.028985:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202689.028987:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187737984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202689.028990:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187738112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202689.029003:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202689.029012:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202689/real 1788202689] req@ffff8e828d723800 x1875062187738112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202689.029024:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187738112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202689.029033:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202689.029036:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202689.029040:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187738112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202689.029047:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187738240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202689.029066:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202689.029075:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202689/real 1788202689] req@ffff8e828d722680 x1875062187738240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202714 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202689.029083:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187738240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202714 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202689.029089:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202689.029091:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202689.029093:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187738240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202689.732546:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202690.756525:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202691.780480:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202692.804443:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202693.828535:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202694.148420:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202694.148427:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202694.148430:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202694.148439:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4149 00000100:00080000:2.0:1788202694.148447:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202694.148453:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202694.148484:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187738368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202694.148504:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202694.148506:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202694.148507:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202694.148510:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4149 00000100:00080000:2.0:1788202694.148512:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202694.148515:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202694.148522:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202694.148529:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187738496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202694.148535:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202694.148537:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202694.148538:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202694.148540:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4149 00000100:00080000:2.0:1788202694.148542:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202694.148543:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202694.148547:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202694.148549:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187738368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202694.148554:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187738624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202694.148560:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202694.148562:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202694.148563:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202694.148565:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4149 00000100:00080000:2.0:1788202694.148567:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202694.148569:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202694.148571:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202694.148577:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187738752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202694.148582:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202694.148584:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202694.148585:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202694.148588:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4149 00000100:00080000:2.0:1788202694.148591:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202694.148593:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202694.148595:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202694.148600:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187738880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202694.148616:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202694.148655:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202694/real 1788202694] req@ffff8e82894e4a80 x1875062187738368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202694.148672:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187738368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202694.148682:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202694.148686:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202694.148689:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187738368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202694.148712:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187738496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202694.148732:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202694.148743:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202694/real 1788202694] req@ffff8e82894e6300 x1875062187738496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202694.148759:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187738496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202694.148767:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202694.148769:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202694.148773:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187738496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202694.148778:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187738624:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202694.148796:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202694.148808:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202694/real 1788202694] req@ffff8e82894e7800 x1875062187738624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202694.148819:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187738624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202694.148826:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202694.148829:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202694.148832:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187738624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202694.148838:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187738752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202694.148856:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202694.148867:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202694/real 1788202694] req@ffff8e82894e4e00 x1875062187738752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202694.148885:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187738752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202694.148898:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202694.148901:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202694.148904:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187738752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202694.148909:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187738880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202694.148927:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202694.148937:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202694/real 1788202694] req@ffff8e82894e6680 x1875062187738880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202719 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202694.148947:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187738880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202719 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202694.148954:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202694.148957:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202694.148960:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187738880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202694.852486:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202695.876492:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202696.900475:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202697.924514:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202698.948487:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202699.268463:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202699.268477:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202699.268482:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202699.268495:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4154 00000100:00080000:0.0:1788202699.268509:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202699.268517:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202699.268564:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187739008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202699.268604:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202699.268612:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202699.268615:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202699.268620:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4154 00000100:00080000:0.0:1788202699.268625:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202699.268630:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202699.268642:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202699.268651:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187739008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202699.268657:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187739136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202699.268673:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202699.268677:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202699.268679:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202699.268685:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4154 00000100:00080000:0.0:1788202699.268689:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202699.268694:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202699.268700:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202699.268712:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187739264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202699.268722:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202699.268726:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202699.268728:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202699.268736:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4154 00000800:00000100:1.0:1788202699.268737:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202699.268743:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202699.268749:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202699.268754:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202699.268768:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187739392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202699.268778:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202699.268782:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202699.268785:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788202699.268788:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202699/real 1788202699] req@ffff8e828793fb80 x1875062187739008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202699.268792:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4154 00000100:00080000:0.0:1788202699.268798:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202699.268804:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202699.268809:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202699.268811:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187739008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202699.268823:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788202699.268825:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187739520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202699.268830:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202699.268837:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187739008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202699.268865:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187739136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202699.268882:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202699.268890:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202699/real 1788202699] req@ffff8e828793d500 x1875062187739136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202699.268899:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187739136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202699.268904:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202699.268906:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202699.268909:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187739136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202699.268913:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187739264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202699.268923:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202699.268929:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202699/real 1788202699] req@ffff8e828793d880 x1875062187739264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202699.268937:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187739264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202699.268942:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202699.268944:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202699.268946:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187739264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202699.268949:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187739392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202699.268961:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202699.268968:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202699/real 1788202699] req@ffff8e828793c380 x1875062187739392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202699.268975:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187739392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202699.268980:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202699.268982:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202699.268984:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187739392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202699.268987:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187739520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202699.268997:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202699.269002:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202699/real 1788202699] req@ffff8e828793ca80 x1875062187739520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202724 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202699.269009:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187739520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202724 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202699.269015:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202699.269016:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202699.269018:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187739520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202699.972515:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202700.996514:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202702.020545:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202703.044473:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202704.068546:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202704.388490:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202704.388504:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202704.388510:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202704.388525:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4159 00000100:00080000:1.0:1788202704.388539:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202704.388548:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202704.388598:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187739648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202704.388637:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202704.388641:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202704.388644:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202704.388649:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4159 00000100:00080000:1.0:1788202704.388653:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202704.388657:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202704.388669:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202704.388685:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187739776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202704.388698:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788202704.388701:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187739648:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202704.388703:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202704.388708:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202704.388715:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4159 00000100:00080000:1.0:1788202704.388718:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202704.388723:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202704.388729:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202704.388742:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187739904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202704.388759:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202704.388762:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202704.388764:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202704.388768:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4159 00000100:00080000:1.0:1788202704.388782:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202704.388787:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202704.388793:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202704.388804:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187740032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202704.388812:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202704.388816:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202704.388818:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202704.388822:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4159 00000100:00080000:1.0:1788202704.388825:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202704.388829:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202704.388834:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202704.388846:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187740160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202704.388859:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187739776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202704.388902:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187739904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202704.388932:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187740032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202704.388982:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187740160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202705.092460:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202706.116496:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202707.140502:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788202707.460591:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:2.0:1788202707.460649:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788202707.460677:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788202707.460726:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788202707.460741:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788202707.460753:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788202707.460766:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788202707.461087:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202704/real 1788202707] req@ffff8e828d720e00 x1875062187740160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202707.461111:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187740160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202707.461124:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202707.461129:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202707.461133:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187740160:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202707.461141:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202704/real 1788202707] req@ffff8e828d723800 x1875062187739776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202707.461154:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187739776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202707.461163:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202707.461166:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202707.461170:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187739776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202707.461178:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202704/real 1788202707] req@ffff8e828d721500 x1875062187739904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202707.461190:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187739904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202707.461198:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202707.461202:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202707.461205:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187739904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202707.461212:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202704/real 1788202707] req@ffff8e828d720000 x1875062187740032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202707.461224:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187740032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202707.461234:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202707.461238:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202707.461241:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187740032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202707.461247:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202704/real 1788202707] req@ffff8e828d722680 x1875062187739648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202707.461258:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187739648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202707.461266:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202707.461269:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202707.461272:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187739648:192.168.201.121@tcp:250:kworker.0 00010000:00010000:0.0:1788202708.164532:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202709.188530:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202709.508412:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202709.508423:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202709.508427:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202709.508438:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4164 00000100:00080000:1.0:1788202709.508449:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202709.508458:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202709.508493:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187740288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202709.508525:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202709.508529:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202709.508531:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202709.508534:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4164 00000100:00080000:1.0:1788202709.508538:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202709.508541:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202709.508550:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202709.508561:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187740416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202709.508577:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187740288:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202709.508586:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202709.508591:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202709.508594:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202709.508598:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4164 00000100:00080000:1.0:1788202709.508602:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202709.508606:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202709.508612:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202709.508650:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187740544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202709.508655:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788202709.508666:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202709.508670:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202709.508671:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202709.508676:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4164 00000100:00080000:1.0:1788202709.508680:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202709.508684:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202709.508689:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202709.508707:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187740672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202709.508723:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202709/real 1788202709] req@ffff8e828d722680 x1875062187740288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202709.508724:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202709.508732:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202709.508736:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202709.508746:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4164 00000100:00080000:1.0:1788202709.508752:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202709.508756:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202709.508762:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187740288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788202709.508764:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788202709.508778:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1788202709.508784:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187740800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202709.508787:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202709.508798:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187740288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202709.508837:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187740416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202709.508883:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202709.508899:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202709/real 1788202709] req@ffff8e828d720000 x1875062187740416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202709.508912:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187740416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202709.508920:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202709.508924:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202709.508927:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187740416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202709.508934:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187740544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202709.508955:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202709.508968:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202709/real 1788202709] req@ffff8e828d721500 x1875062187740544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202709.508986:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187740544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202709.508994:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202709.508996:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202709.508999:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187740544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202709.509005:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187740672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202709.509028:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202709.509039:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202709/real 1788202709] req@ffff8e828d723800 x1875062187740672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202709.509051:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187740672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202709.509059:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202709.509062:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202709.509065:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187740672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202709.509071:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187740800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202709.509091:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202709.509102:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202709/real 1788202709] req@ffff8e828d720e00 x1875062187740800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202709.509114:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187740800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202709.509128:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202709.509131:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202709.509134:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187740800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202710.212453:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202711.236440:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202712.260498:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202713.284479:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202714.308457:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202714.628454:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202714.628461:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202714.628465:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202714.628474:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4169 00000100:00080000:0.0:1788202714.628484:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202714.628492:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202714.628536:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187740928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202714.628562:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202714.628566:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202714.628568:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202714.628573:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4169 00000100:00080000:0.0:1788202714.628577:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202714.628583:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202714.628585:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187740928:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788202714.628592:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202714.628609:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187741056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202714.628620:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202714.628625:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202714.628627:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202714.628632:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4169 00000100:00080000:0.0:1788202714.628636:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202714.628641:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202714.628648:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788202714.628656:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202714.628664:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187741184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202714.628676:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202714.628680:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202714.628682:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202714.628687:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4169 00000100:00080000:0.0:1788202714.628692:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202714.628697:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202714.628703:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202714.628717:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187741312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202714.628725:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202714.628728:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202714.628730:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202714.628735:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4169 00000100:00080000:0.0:1788202714.628740:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202714.628744:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202714.628748:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202714.628757:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187741440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202714.628784:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202714/real 1788202714] req@ffff8e828793f480 x1875062187740928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202714.628819:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187740928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202714.628944:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202714.628949:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202714.628954:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187740928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202714.629014:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187741056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202714.629039:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202714.629052:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202714/real 1788202714] req@ffff8e828793dc00 x1875062187741056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202714.629064:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187741056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202714.629072:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202714.629076:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202714.629079:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187741056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202714.629085:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187741184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202714.629104:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202714.629114:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202714/real 1788202714] req@ffff8e828793c000 x1875062187741184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202714.629125:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187741184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202714.629133:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202714.629136:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202714.629140:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187741184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202714.629145:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187741312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202714.629163:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202714.629173:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202714/real 1788202714] req@ffff8e828793e300 x1875062187741312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202714.629189:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187741312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202714.629197:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202714.629200:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202714.629203:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187741312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202714.629208:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187741440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202714.629226:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202714.629236:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202714/real 1788202714] req@ffff8e828793c700 x1875062187741440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202714.629247:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187741440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202714.629255:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202714.629258:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202714.629261:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187741440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788202715.332464:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202716.356552:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202717.380528:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202718.404452:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202719.428502:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202719.748414:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202719.748423:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202719.748426:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202719.748435:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4174 00000100:00080000:0.0:1788202719.748444:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202719.748450:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202719.748482:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187741568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202719.748505:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202719.748507:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202719.748509:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202719.748512:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4174 00000100:00080000:0.0:1788202719.748514:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202719.748516:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202719.748524:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202719.748534:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187741696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202719.748545:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187741568:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202719.748548:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202719.748552:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202719.748553:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202719.748556:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4174 00000100:00080000:0.0:1788202719.748561:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202719.748564:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202719.748578:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202719.748593:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187741824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202719.748604:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202719.748608:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202719.748610:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788202719.748616:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202719.748617:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4174 00000100:00080000:0.0:1788202719.748624:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202719.748629:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202719.748633:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202719.748644:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187741952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202719.748651:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202719.748653:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202719.748655:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202719.748658:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4174 00000100:00080000:0.0:1788202719.748661:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788202719.748665:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202719/real 1788202719] req@ffff8e828793f100 x1875062187741568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202719.748666:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202719.748673:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202719.748699:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187742080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202719.748705:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187741568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202719.748728:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202719.748732:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202719.748736:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187741568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202719.748756:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187741696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202719.748774:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202719.748787:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202719/real 1788202719] req@ffff8e828793e680 x1875062187741696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202719.748800:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187741696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202719.748809:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202719.748813:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202719.748816:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187741696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202719.748829:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187741824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202719.748850:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202719.748860:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202719/real 1788202719] req@ffff8e828793f800 x1875062187741824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202719.748872:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187741824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202719.748879:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202719.748883:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202719.748886:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187741824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202719.748893:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187741952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202719.748914:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202719.748923:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202719/real 1788202719] req@ffff8e828793ce00 x1875062187741952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202719.748931:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187741952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202719.748936:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202719.748938:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202719.748941:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187741952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202719.748944:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187742080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202719.748955:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202719.748961:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202719/real 1788202719] req@ffff8e828793c700 x1875062187742080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202719.748968:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187742080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202719.748972:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202719.748974:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202719.748976:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187742080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202720.452444:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202721.476531:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202722.500445:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202723.524546:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202724.548488:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202724.868467:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202724.868477:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202724.868482:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202724.868493:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4179 00000100:00080000:2.0:1788202724.868506:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202724.868513:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202724.868565:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187742208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202724.868600:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202724.868603:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202724.868605:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202724.868609:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4179 00000100:00080000:2.0:1788202724.868612:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202724.868615:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202724.868625:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202724.868635:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187742336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202724.868644:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202724.868647:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202724.868649:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202724.868652:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4179 00000100:00080000:2.0:1788202724.868656:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202724.868659:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202724.868662:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202724.868669:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187742464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202724.868675:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202724.868679:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202724.868680:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202724.868684:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4179 00000100:00080000:2.0:1788202724.868687:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202724.868689:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202724.868693:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202724.868701:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187742208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202724.868708:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187742592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202724.868719:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202724.868721:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202724.868722:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202724.868727:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4179 00000100:00080000:2.0:1788202724.868733:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202724.868737:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202724.868740:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202724.868755:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187742720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202724.868779:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202724.868829:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202724/real 1788202724] req@ffff8e82894e7480 x1875062187742208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202724.868850:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187742208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202724.868862:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202724.868867:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202724.868872:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187742208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202724.868880:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187742336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202724.868905:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202724.868917:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202724/real 1788202724] req@ffff8e82894e6a00 x1875062187742336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202724.868926:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187742336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202724.868931:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202724.868934:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202724.868936:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187742336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202724.868940:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187742464:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202724.868953:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202724.868959:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202724/real 1788202724] req@ffff8e82894e4000 x1875062187742464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202724.868966:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187742464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202724.868976:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202724.868978:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202724.868980:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187742464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202724.869010:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187742592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202724.869024:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202724.869031:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202724/real 1788202724] req@ffff8e82894e5880 x1875062187742592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202724.869038:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187742592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202724.869047:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202724.869049:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202724.869051:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187742592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202724.869056:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187742720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202724.869066:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202724.869072:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202724/real 1788202724] req@ffff8e82894e6680 x1875062187742720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202724.869079:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187742720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202724.869084:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202724.869086:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202724.869088:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187742720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202725.572509:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202726.596570:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202727.620525:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202728.644519:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202729.668477:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202729.988433:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202729.988444:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202729.988448:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202729.988459:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4185 00000100:00080000:0.0:1788202729.988471:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202729.988479:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202729.988535:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187742848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202729.988560:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202729.988563:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202729.988565:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202729.988569:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4185 00000100:00080000:0.0:1788202729.988573:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202729.988577:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202729.988586:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202729.988596:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187742976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202729.988605:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202729.988607:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202729.988609:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202729.988611:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4185 00000100:00080000:0.0:1788202729.988613:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202729.988615:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202729.988618:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202729.988622:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187742848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202729.988629:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187743104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202729.988642:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202729.988646:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202729.988648:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202729.988653:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4185 00000100:00080000:0.0:1788202729.988657:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202729.988661:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202729.988667:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202729.988681:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187743232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202729.988683:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202729.988696:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202729.988700:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202729.988702:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202729.988708:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4185 00000100:00080000:0.0:1788202729.988713:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202729.988718:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202729.988722:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202729/real 1788202729] req@ffff8e828793e300 x1875062187742848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788202729.988725:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202729.988741:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187743360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202729.988744:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187742848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202729.988767:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202729.988771:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202729.988774:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187742848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202729.988781:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187742976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202729.988797:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202729.988805:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202729/real 1788202729] req@ffff8e828793c000 x1875062187742976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202729.988813:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187742976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202729.988819:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202729.988821:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202729.988824:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187742976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202729.988840:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187743104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202729.988851:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202729.988858:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202729/real 1788202729] req@ffff8e828793dc00 x1875062187743104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202729.988865:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187743104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202729.988871:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202729.988873:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202729.988875:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187743104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202729.988878:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187743232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202729.988889:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202729.988895:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202729/real 1788202729] req@ffff8e828793f480 x1875062187743232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202729.988902:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187743232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202729.988907:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202729.988909:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202729.988911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187743232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202729.988914:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187743360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202729.988923:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202729.988929:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202729/real 1788202729] req@ffff8e828793ca80 x1875062187743360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202729.988935:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187743360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202729.988941:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202729.988943:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202729.988945:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187743360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788202730.692523:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202731.716541:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202732.740554:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202733.764466:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202734.788535:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202735.108427:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202735.108438:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202735.108443:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202735.108456:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4190 00000100:00080000:2.0:1788202735.108467:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202735.108474:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202735.108514:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187743488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202735.108549:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202735.108554:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202735.108555:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202735.108560:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4190 00000100:00080000:2.0:1788202735.108564:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202735.108569:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202735.108580:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202735.108595:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187743616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202735.108607:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187743488:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202735.108610:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202735.108616:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202735.108618:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202735.108623:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4190 00000100:00080000:2.0:1788202735.108627:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202735.108631:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202735.108637:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202735.108649:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187743744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202735.108659:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202735.108663:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202735.108664:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788202735.108670:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788202735.108674:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4190 00000100:00080000:2.0:1788202735.108680:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202735.108685:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202735.108691:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202735.108705:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187743872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202735.108715:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788202735.108719:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202735/real 1788202735] req@ffff8e82894e4e00 x1875062187743488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202735.108721:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202735.108736:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202735.108744:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4190 00000100:00100000:3.0:1788202735.108745:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187743488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202735.108754:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202735.108758:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202735.108763:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:3.0:1788202735.108763:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202735.108769:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187743488:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788202735.108775:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202735.108792:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187744000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202735.108792:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187743616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202735.108812:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202735.108831:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202735/real 1788202735] req@ffff8e82894e7800 x1875062187743616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202735.108843:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187743616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202735.108849:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202735.108851:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202735.108854:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187743616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202735.108858:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187743744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202735.108871:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202735.108878:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202735/real 1788202735] req@ffff8e82894e6300 x1875062187743744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202735.108886:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187743744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202735.108891:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202735.108893:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202735.108895:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187743744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202735.108898:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187743872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202735.108909:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202735.108915:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202735/real 1788202735] req@ffff8e82894e4a80 x1875062187743872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202735.108922:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187743872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202735.108927:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202735.108929:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202735.108931:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187743872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202735.108940:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187744000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202735.108950:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202735.108956:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202735/real 1788202735] req@ffff8e82894e7100 x1875062187744000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202760 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202735.108964:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187744000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202760 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202735.108969:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202735.108970:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202735.108972:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187744000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202735.812504:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202736.836464:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202737.860457:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202738.884546:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202739.908413:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202740.228463:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202740.228475:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202740.228481:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202740.228491:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4195 00000100:00080000:2.0:1788202740.228503:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202740.228511:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202740.228552:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187744128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202740.228579:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202740.228583:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202740.228586:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202740.228590:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4195 00000100:00080000:2.0:1788202740.228594:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202740.228598:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202740.228607:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202740.228622:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187744256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202740.228636:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187744128:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202740.228655:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202740.228661:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202740.228664:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202740.228669:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4195 00000100:00080000:2.0:1788202740.228674:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202740.228679:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202740.228685:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202740.228698:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187744384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788202740.228709:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788202740.228710:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202740.228717:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202740.228719:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202740.228724:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4195 00000100:00080000:2.0:1788202740.228728:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202740.228733:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202740.228739:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202740.228752:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187744512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202740.228760:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202740/real 1788202740] req@ffff8e82894e5c00 x1875062187744128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202740.228767:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202740.228774:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202740.228776:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788202740.228781:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187744128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202740.228784:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4195 00000100:00080000:2.0:1788202740.228793:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202740.228797:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202740.228801:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:3.0:1788202740.228804:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202740.228810:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187744128:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788202740.228812:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202740.228829:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187744640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202740.228835:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187744256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202740.228869:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202740.228883:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202740/real 1788202740] req@ffff8e82894e4380 x1875062187744256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202740.228896:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187744256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202740.228912:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202740.228915:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202740.228919:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187744256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202740.228925:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187744384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202740.228943:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202740.228951:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202740/real 1788202740] req@ffff8e82894e5f80 x1875062187744384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202740.228967:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187744384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202740.228976:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202740.228980:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202740.228984:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187744384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202740.228990:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187744512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202740.229012:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202740.229024:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202740/real 1788202740] req@ffff8e82894e6d80 x1875062187744512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202740.229032:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187744512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202740.229038:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202740.229040:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202740.229050:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187744512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202740.229062:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187744640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202740.229077:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202740.229084:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202740/real 1788202740] req@ffff8e82894e7100 x1875062187744640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202765 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202740.229092:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187744640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202765 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202740.229097:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202740.229099:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202740.229101:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187744640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202740.932524:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202741.956470:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202742.980446:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202744.004434:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202745.028553:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202745.348449:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202745.348459:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202745.348464:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202745.348474:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4200 00000100:00080000:0.0:1788202745.348484:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202745.348492:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202745.348530:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187744768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202745.348560:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202745.348564:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202745.348567:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202745.348572:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4200 00000100:00080000:0.0:1788202745.348576:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202745.348580:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202745.348589:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202745.348603:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187744896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202745.348615:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202745.348619:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202745.348621:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202745.348626:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4200 00000100:00080000:0.0:1788202745.348630:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202745.348634:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202745.348639:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202745.348654:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187745024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202745.348672:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187744768:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202745.348676:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202745.348686:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202745.348688:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202745.348694:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4200 00000100:00080000:0.0:1788202745.348698:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202745.348705:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202745.348711:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202745.348736:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187745152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202745.348747:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202745.348752:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202745.348754:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202745.348763:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4200 00000800:00000100:3.0:1788202745.348764:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202745.348773:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202745.348784:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202745.348791:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202745.348809:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187745280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788202745.348821:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202745/real 1788202745] req@ffff8e828793c380 x1875062187744768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202745.348839:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187744768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202745.348849:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202745.348854:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202745.348858:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187744768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202745.348865:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187744896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202745.348881:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202745.348891:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202745/real 1788202745] req@ffff8e828793d880 x1875062187744896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202745.348899:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187744896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202745.348904:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202745.348906:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202745.348909:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187744896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202745.348933:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187745024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788202745.348945:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202745.348951:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202745/real 1788202745] req@ffff8e828793d500 x1875062187745024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202745.348959:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187745024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202745.348964:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202745.348966:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202745.348968:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187745024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202745.348971:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187745152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202745.348982:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202745.348988:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202745/real 1788202745] req@ffff8e828793fb80 x1875062187745152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202745.348995:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187745152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202745.349001:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202745.349003:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202745.349005:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187745152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202745.349009:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187745280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788202745.349019:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202745.349025:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202745/real 1788202745] req@ffff8e828793ca80 x1875062187745280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202745.349031:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187745280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202745.349036:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202745.349038:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202745.349040:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187745280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788202746.052472:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202747.076491:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202748.100484:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202749.124485:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202750.148511:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202750.468473:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202750.468489:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202750.468495:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202750.468509:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4205 00000100:00080000:2.0:1788202750.468525:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202750.468534:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202750.468576:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187745408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202750.468599:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202750.468602:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202750.468604:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202750.468609:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4205 00000100:00080000:2.0:1788202750.468612:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202750.468615:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202750.468625:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202750.468631:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187745408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202750.468641:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187745536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202750.468653:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202750.468656:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202750.468657:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202750.468660:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4205 00000100:00080000:2.0:1788202750.468662:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202750.468665:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202750.468671:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202750.468685:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187745664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202750.468697:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202750.468702:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202750.468704:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788202750.468710:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788202750.468713:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4205 00000100:00080000:2.0:1788202750.468720:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202750.468726:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202750.468732:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202750.468744:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187745792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202750.468751:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202750.468754:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202750.468761:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788202750.468763:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202750/real 1788202750] req@ffff8e82894e4a80 x1875062187745408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202750.468768:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4205 00000100:00080000:2.0:1788202750.468774:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202750.468781:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202750.468789:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202750.468799:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187745920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202750.468805:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187745408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202750.468822:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202750.468828:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202750.468834:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187745408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202750.468861:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187745536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202750.468881:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202750.468891:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202750/real 1788202750] req@ffff8e82894e6300 x1875062187745536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202750.468901:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187745536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202750.468908:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202750.468911:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202750.468914:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187745536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202750.468918:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187745664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202750.468931:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202750.468938:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202750/real 1788202750] req@ffff8e82894e7800 x1875062187745664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202750.468946:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187745664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202750.468957:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202750.468959:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202750.468962:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187745664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202750.468965:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187745792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202750.468977:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202750.468982:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202750/real 1788202750] req@ffff8e82894e4e00 x1875062187745792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202750.468990:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187745792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202750.468995:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202750.468997:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202750.468999:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187745792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202750.469002:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187745920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202750.469013:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202750.469019:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202750/real 1788202750] req@ffff8e82894e6680 x1875062187745920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202750.469027:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187745920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202750.469032:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202750.469034:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202750.469037:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187745920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202751.172530:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202752.196494:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202753.220526:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202754.244557:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202755.268478:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202755.588473:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202755.588489:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202755.588495:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202755.588508:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4210 00000100:00080000:0.0:1788202755.588524:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202755.588535:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202755.588586:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187746048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202755.588618:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202755.588622:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202755.588624:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202755.588629:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4210 00000100:00080000:0.0:1788202755.588633:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202755.588638:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202755.588650:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202755.588666:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187746048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202755.588668:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187746176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202755.588683:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202755.588687:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202755.588689:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202755.588693:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4210 00000100:00080000:0.0:1788202755.588697:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202755.588702:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202755.588709:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202755.588722:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187746304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202755.588733:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202755.588736:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202755.588738:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788202755.588744:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202755.588749:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4210 00000100:00080000:0.0:1788202755.588755:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202755.588760:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202755.588767:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202755.588782:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187746432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202755.588793:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788202755.588795:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202755/real 1788202755] req@ffff8e828793f480 x1875062187746048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202755.588799:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202755.588804:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202755.588812:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4210 00000100:00080000:0.0:1788202755.588817:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202755.588817:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187746048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202755.588827:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:1.0:1788202755.588833:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788202755.588835:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788202755.588838:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202755.588844:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187746048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202755.588854:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187746560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202755.588896:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187746176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202755.588921:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202755.588934:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202755/real 1788202755] req@ffff8e828793dc00 x1875062187746176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202755.588947:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187746176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202755.588956:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202755.588960:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202755.588962:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187746176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202755.588967:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187746304:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202755.588984:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202755.588991:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202755/real 1788202755] req@ffff8e828793c000 x1875062187746304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202755.588999:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187746304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202755.589004:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202755.589006:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202755.589008:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187746304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202755.589012:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187746432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202755.589023:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202755.589029:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202755/real 1788202755] req@ffff8e828793e300 x1875062187746432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202755.589036:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187746432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202755.589041:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202755.589043:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202755.589045:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187746432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202755.589048:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187746560:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202755.589076:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202755.589084:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202755/real 1788202755] req@ffff8e828793c700 x1875062187746560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202755.589097:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187746560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202755.589102:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202755.589104:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202755.589105:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187746560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202756.292530:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202757.316495:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202758.340469:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202759.364476:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202760.388527:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202760.708476:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202760.708491:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202760.708497:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202760.708511:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4215 00000100:00080000:2.0:1788202760.708525:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202760.708535:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202760.708602:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187746688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202760.708641:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202760.708646:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202760.708648:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202760.708653:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4215 00000100:00080000:2.0:1788202760.708658:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202760.708662:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202760.708674:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202760.708688:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187746816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202760.708700:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202760.708717:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202760.708719:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202760.708724:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4215 00000100:00080000:2.0:1788202760.708728:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202760.708732:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202760.708738:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202760.708749:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187746944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202760.708759:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202760.708763:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202760.708765:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202760.708769:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4215 00000100:00080000:2.0:1788202760.708772:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202760.708776:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202760.708782:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202760.708802:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187747072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202760.708809:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187746688:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202760.708821:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202760.708826:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202760.708828:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202760.708833:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4215 00000100:00080000:2.0:1788202760.708838:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202760.708843:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202760.708850:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202760.708865:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187747200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202760.708884:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202760.708959:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202760/real 1788202760] req@ffff8e82894e5880 x1875062187746688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202760.709000:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187746688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202760.709013:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202760.709018:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202760.709023:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187746688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202760.709032:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187746816:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202760.709059:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202760.709076:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202760/real 1788202760] req@ffff8e82894e4000 x1875062187746816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202760.709090:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187746816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202760.709100:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202760.709104:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202760.709108:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187746816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202760.709114:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187746944:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202760.709132:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202760.709140:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202760/real 1788202760] req@ffff8e82894e6a00 x1875062187746944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202760.709148:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187746944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202760.709154:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202760.709157:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202760.709159:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187746944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202760.709182:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187747072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202760.709201:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202760.709212:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202760/real 1788202760] req@ffff8e82894e7480 x1875062187747072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202760.709235:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187747072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202760.709245:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202760.709247:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202760.709251:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187747072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202760.709256:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187747200:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202760.709273:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202760.709281:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202760/real 1788202760] req@ffff8e82894e6680 x1875062187747200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202760.709288:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187747200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202760.709294:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202760.709296:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202760.709298:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187747200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788202761.412395:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202762.436513:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202763.460514:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202764.484531:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202765.508496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202765.828464:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202765.828479:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202765.828485:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202765.828498:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4220 00000100:00080000:3.0:1788202765.828513:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202765.828523:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202765.828577:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187747328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202765.828611:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202765.828616:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202765.828618:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202765.828623:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4220 00000100:00080000:3.0:1788202765.828627:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202765.828632:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202765.828644:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202765.828659:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187747456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202765.828674:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187747328:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202765.828691:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202765.828694:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202765.828696:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202765.828700:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4220 00000100:00080000:3.0:1788202765.828703:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202765.828707:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202765.828712:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202765.828728:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187747584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202765.828740:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202765.828744:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202765.828745:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202765.828750:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4220 00000100:00080000:3.0:1788202765.828754:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202765.828759:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202765.828766:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788202765.828772:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202765.828784:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187747712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202765.828796:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202765.828801:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202765.828803:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202765.828808:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4220 00000100:00080000:3.0:1788202765.828813:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202765.828817:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202765.828823:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202765.828838:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187747840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202765.828845:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202765/real 1788202765] req@ffff8e8289538a80 x1875062187747328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202765.828872:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187747328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202765.828899:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202765.828904:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202765.828909:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187747328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202765.828939:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187747456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202765.828967:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202765.828982:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202765/real 1788202765] req@ffff8e828953a680 x1875062187747456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202765.828994:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187747456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202765.829000:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202765.829003:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202765.829005:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187747456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202765.829010:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187747584:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202765.829033:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202765.829042:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202765/real 1788202765] req@ffff8e828953b100 x1875062187747584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202765.829049:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187747584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202765.829055:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202765.829057:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202765.829059:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187747584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202765.829062:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187747712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202765.829074:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202765.829080:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202765/real 1788202765] req@ffff8e8289539500 x1875062187747712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202765.829093:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187747712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202765.829098:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202765.829099:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202765.829101:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187747712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202765.829104:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187747840:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202765.829123:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202765.829129:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202765/real 1788202765] req@ffff8e8289539880 x1875062187747840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202765.829136:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187747840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202765.829141:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202765.829143:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202765.829144:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187747840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202766.532515:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202767.556481:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202768.580514:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202769.604521:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202770.628465:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202770.948456:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202770.948468:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202770.948473:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202770.948483:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4226 00000100:00080000:2.0:1788202770.948495:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202770.948503:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202770.948557:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187747968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202770.948588:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202770.948593:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202770.948595:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202770.948601:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4226 00000100:00080000:2.0:1788202770.948607:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202770.948613:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202770.948626:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202770.948633:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187747968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202770.948648:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187748096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202770.948664:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202770.948669:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202770.948671:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202770.948684:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4226 00000100:00080000:2.0:1788202770.948691:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202770.948696:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202770.948705:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202770.948720:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187748224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202770.948731:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202770.948735:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202770.948737:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202770.948742:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4226 00000100:00080000:2.0:1788202770.948747:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202770.948754:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202770.948757:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187748096:192.168.201.121@tcp:38:kworker.0 00000008:00100000:2.0:1788202770.948766:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202770.948803:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187748352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202770.948818:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788202770.948819:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187748224:192.168.201.121@tcp:38:kworker.0 00000100:00080000:2.0:1788202770.948830:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202770.948834:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202770.948841:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4226 00000100:00080000:2.0:1788202770.948847:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202770.948854:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202770.948860:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202770.948875:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187748352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202770.948891:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187748480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202770.948928:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187748480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202771.652563:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202772.676500:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202773.700451:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788202774.020671:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788202774.020755:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788202774.020778:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202774.020844:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202774.020857:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202774.020866:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202774.020876:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788202774.021008:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202770/real 1788202774] req@ffff8e82894e7100 x1875062187748480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202774.021045:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187748480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202774.021064:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202774.021072:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202774.021078:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187748480:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202774.021088:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202770/real 1788202774] req@ffff8e82894e4a80 x1875062187748352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202774.021103:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187748352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202774.021113:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202774.021118:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202774.021121:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187748352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202774.021129:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202770/real 1788202774] req@ffff8e82894e7800 x1875062187748096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202774.021141:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187748096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202774.021156:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202774.021161:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202774.021164:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187748096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202774.021172:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202770/real 1788202774] req@ffff8e82894e6300 x1875062187748224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202774.021185:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187748224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202774.021192:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202774.021196:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202774.021199:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187748224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202774.021205:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202770/real 1788202774] req@ffff8e82894e4e00 x1875062187747968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202774.021218:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187747968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202774.021226:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202774.021229:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202774.021232:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187747968:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788202774.724501:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202775.748563:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202776.068401:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202776.068411:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202776.068416:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202776.068426:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4231 00000100:00080000:2.0:1788202776.068439:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202776.068446:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202776.068491:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187748608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202776.068517:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202776.068522:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202776.068524:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202776.068528:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4231 00000100:00080000:2.0:1788202776.068533:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202776.068538:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202776.068549:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202776.068560:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187748608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202776.068568:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187748736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202776.068582:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202776.068597:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202776.068599:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202776.068603:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4231 00000100:00080000:2.0:1788202776.068608:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202776.068613:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202776.068618:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788202776.068623:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202776.068636:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187748864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202776.068647:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202776.068651:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202776.068653:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202776.068658:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4231 00000100:00080000:2.0:1788202776.068668:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202776.068672:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202776/real 1788202776] req@ffff8e82894e6d80 x1875062187748608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202776.068677:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202776.068685:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202776.068715:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187748608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202776.068722:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187748992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202776.068737:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202776.068741:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202776.068746:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:2.0:1788202776.068748:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202776.068752:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788202776.068754:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187748608:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202776.068761:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4231 00000100:00080000:2.0:1788202776.068767:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202776.068772:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202776.068779:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202776.068788:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187748736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202776.068795:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187749120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202776.068822:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202776.068840:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202776/real 1788202776] req@ffff8e82894e5f80 x1875062187748736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202776.068854:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187748736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202776.068863:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202776.068867:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202776.068871:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187748736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202776.068877:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187748864:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202776.068899:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202776.068912:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202776/real 1788202776] req@ffff8e82894e4380 x1875062187748864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202776.068924:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187748864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202776.068938:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202776.068942:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202776.068946:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187748864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202776.068952:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187748992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202776.068974:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202776.068986:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202776/real 1788202776] req@ffff8e82894e5c00 x1875062187748992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202776.068998:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187748992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202776.069006:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202776.069009:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202776.069012:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187748992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202776.069030:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187749120:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202776.069051:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202776.069063:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202776/real 1788202776] req@ffff8e82894e6d80 x1875062187749120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202801 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202776.069076:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187749120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202801 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202776.069082:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202776.069084:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202776.069087:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187749120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788202776.772521:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202777.796512:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202778.820451:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202779.844487:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202780.868471:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202781.188449:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202781.188460:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202781.188465:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202781.188477:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4236 00000100:00080000:3.0:1788202781.188490:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202781.188499:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202781.188544:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187749248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202781.188568:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202781.188572:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202781.188573:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202781.188577:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4236 00000100:00080000:3.0:1788202781.188580:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202781.188584:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202781.188593:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202781.188606:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187749248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202781.188607:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187749376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202781.188640:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202781.188642:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202781.188644:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202781.188647:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4236 00000100:00080000:3.0:1788202781.188650:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202781.188654:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202781.188658:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202781.188670:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187749504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202781.188677:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202781.188680:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202781.188682:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202781.188685:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4236 00000100:00080000:3.0:1788202781.188688:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202781.188690:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202781.188693:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202781.188700:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187749632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202781.188707:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202781.188709:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202781.188710:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202781.188713:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4236 00000100:00080000:3.0:1788202781.188715:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202781.188720:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:1.0:1788202781.188722:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:3.0:1788202781.188725:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202781.188734:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187749760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202781.188787:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202781/real 1788202781] req@ffff8e828953bb80 x1875062187749248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202781.188812:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187749248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202781.188826:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202781.188831:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202781.188836:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187749248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202781.188868:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187749376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202781.188896:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202781.188908:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202781/real 1788202781] req@ffff8e8289538380 x1875062187749376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202781.188917:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187749376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202781.188923:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202781.188926:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202781.188928:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187749376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202781.188943:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187749504:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202781.188958:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202781.188966:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202781/real 1788202781] req@ffff8e8289539f80 x1875062187749504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202781.188975:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187749504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202781.188980:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202781.188983:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202781.188985:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187749504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202781.188988:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187749632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202781.189000:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202781.189006:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202781/real 1788202781] req@ffff8e8289539880 x1875062187749632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202781.189014:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187749632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202781.189019:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202781.189021:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202781.189023:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187749632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202781.189026:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187749760:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202781.189036:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202781.189042:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202781/real 1788202781] req@ffff8e8289539500 x1875062187749760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202806 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202781.189050:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187749760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202806 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202781.189055:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202781.189057:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202781.189059:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187749760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202781.892481:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202782.916482:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202783.940505:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202784.964463:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202785.988434:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202786.308446:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202786.308458:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202786.308463:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202786.308476:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4241 00000100:00080000:3.0:1788202786.308488:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202786.308497:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202786.308553:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187749888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202786.308592:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202786.308597:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202786.308599:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202786.308605:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4241 00000100:00080000:3.0:1788202786.308609:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202786.308614:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202786.308628:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202786.308648:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187750016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202786.308653:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187749888:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202786.308660:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202786.308665:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202786.308667:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202786.308671:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4241 00000100:00080000:3.0:1788202786.308675:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202786.308678:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202786.308684:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202786.308695:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187750144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202786.308702:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202786.308705:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202786.308706:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202786.308709:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4241 00000100:00080000:3.0:1788202786.308724:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202786.308727:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202786.308731:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202786.308738:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187750272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202786.308746:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788202786.308749:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202786.308753:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202786.308755:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202786.308758:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4241 00000100:00080000:3.0:1788202786.308761:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202786.308764:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202786.308768:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202786.308808:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202786/real 1788202786] req@ffff8e828953b100 x1875062187749888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202786.308838:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187749888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202786.308852:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202786.308862:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788202786.308863:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187750400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202786.308872:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187749888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202786.308913:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187750016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202786.308952:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202786.308966:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202786/real 1788202786] req@ffff8e828953a680 x1875062187750016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202786.308979:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187750016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202786.308988:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202786.308993:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202786.308996:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187750016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202786.309002:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187750144:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202786.309020:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202786.309032:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202786/real 1788202786] req@ffff8e8289538a80 x1875062187750144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202786.309043:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187750144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202786.309057:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202786.309062:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202786.309065:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187750144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202786.309072:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187750272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202786.309092:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202786.309103:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202786/real 1788202786] req@ffff8e8289539500 x1875062187750272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202786.309114:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187750272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202786.309123:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202786.309126:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202786.309130:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187750272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202786.309136:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187750400:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202786.309156:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202786.309166:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202786/real 1788202786] req@ffff8e8289539880 x1875062187750400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202786.309178:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187750400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202786.309187:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202786.309191:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202786.309195:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187750400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202787.012550:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202788.036487:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202789.060526:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202790.084520:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202791.108551:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202791.428449:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202791.428459:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202791.428462:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202791.428471:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4246 00000100:00080000:3.0:1788202791.428480:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202791.428486:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202791.428521:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187750528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202791.428547:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202791.428552:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202791.428553:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202791.428556:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4246 00000100:00080000:3.0:1788202791.428558:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202791.428561:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202791.428569:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202791.428579:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187750656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202791.428586:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202791.428589:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202791.428590:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202791.428592:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4246 00000100:00080000:3.0:1788202791.428595:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202791.428597:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202791.428600:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202791.428603:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187750528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202791.428609:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187750784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202791.428618:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202791.428620:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202791.428621:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202791.428624:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4246 00000100:00080000:3.0:1788202791.428626:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202791.428628:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202791.428631:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202791.428640:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187750912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202791.428645:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202791.428648:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202791.428649:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202791.428651:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4246 00000100:00080000:3.0:1788202791.428654:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202791.428657:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202791.428660:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788202791.428671:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202791.428675:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187751040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202791.428732:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202791/real 1788202791] req@ffff8e8289539f80 x1875062187750528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202791.428757:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187750528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202791.428768:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202791.428771:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202791.428774:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187750528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202791.428779:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187750656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202791.428799:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202791.428808:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202791/real 1788202791] req@ffff8e8289538380 x1875062187750656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202791.428821:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187750656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202791.428827:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202791.428829:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202791.428831:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187750656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202791.428849:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187750784:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202791.428861:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202791.428867:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202791/real 1788202791] req@ffff8e828953bb80 x1875062187750784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202791.428874:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187750784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202791.428879:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202791.428881:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202791.428883:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187750784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202791.428887:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187750912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202791.428897:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202791.428902:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202791/real 1788202791] req@ffff8e8289539880 x1875062187750912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202791.428910:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187750912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202791.428919:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202791.428921:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202791.428923:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187750912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202791.428926:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187751040:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202791.428935:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202791.428940:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202791/real 1788202791] req@ffff8e8289539500 x1875062187751040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202791.428947:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187751040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202791.428951:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202791.428953:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202791.428955:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187751040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788202792.132409:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202793.156514:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202794.180468:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202795.204509:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202796.228493:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202796.548493:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202796.548504:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202796.548509:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202796.548520:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4251 00000100:00080000:2.0:1788202796.548533:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202796.548540:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202796.548584:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187751168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202796.548617:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202796.548620:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202796.548621:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202796.548625:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4251 00000100:00080000:2.0:1788202796.548627:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202796.548630:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202796.548639:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202796.548649:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187751296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202796.548657:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202796.548660:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202796.548662:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788202796.548666:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187751168:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202796.548667:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4251 00000100:00080000:2.0:1788202796.548671:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202796.548677:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202796.548681:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202796.548687:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187751424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202796.548693:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202796.548695:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202796.548697:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202796.548700:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4251 00000100:00080000:2.0:1788202796.548703:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202796.548706:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202796.548709:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202796.548718:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187751552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202796.548724:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202796.548727:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202796.548728:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202796.548731:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4251 00000100:00080000:2.0:1788202796.548734:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202796.548736:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202796.548739:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202796.548746:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187751680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202796.548761:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202796.548827:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202796/real 1788202796] req@ffff8e82894e4e00 x1875062187751168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202796.548845:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187751168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202796.548854:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202796.548859:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202796.548863:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187751168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202796.548887:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187751296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202796.548906:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202796.548916:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202796/real 1788202796] req@ffff8e82894e6300 x1875062187751296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202796.548924:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187751296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202796.548930:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202796.548932:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202796.548934:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187751296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202796.548938:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187751424:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202796.548949:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202796.548956:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202796/real 1788202796] req@ffff8e82894e7800 x1875062187751424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202796.548964:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187751424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202796.548969:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202796.548970:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202796.548972:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187751424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202796.548976:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187751552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202796.548987:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202796.548993:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202796/real 1788202796] req@ffff8e82894e4a80 x1875062187751552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202796.549001:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187751552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202796.549006:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202796.549008:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202796.549010:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187751552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202796.549013:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187751680:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202796.549023:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202796.549029:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202796/real 1788202796] req@ffff8e82894e7100 x1875062187751680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202796.549037:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187751680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202796.549042:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202796.549044:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202796.549045:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187751680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202797.252520:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202798.276452:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202799.300514:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202800.324481:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202801.348478:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202801.668459:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202801.668470:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202801.668475:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202801.668487:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4256 00000100:00080000:0.0:1788202801.668501:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202801.668511:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202801.668567:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187751808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202801.668596:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202801.668599:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202801.668601:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202801.668606:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4256 00000100:00080000:0.0:1788202801.668610:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202801.668614:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202801.668625:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202801.668629:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187751808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202801.668641:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187751936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202801.668650:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202801.668654:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202801.668656:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202801.668660:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4256 00000100:00080000:0.0:1788202801.668674:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202801.668684:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202801.668690:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202801.668705:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187752064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202801.668715:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788202801.668715:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202801.668722:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202801.668723:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202801.668727:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4256 00000100:00080000:0.0:1788202801.668731:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202801.668734:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202801.668739:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202801.668749:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187752192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202801.668756:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202801.668759:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202801.668761:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202801.668764:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4256 00000100:00080000:0.0:1788202801.668768:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202801.668769:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202801/real 1788202801] req@ffff8e828793ce00 x1875062187751808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202801.668773:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202801.668781:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202801.668792:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187752320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202801.668802:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187751808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202801.668813:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202801.668817:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202801.668821:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187751808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202801.668844:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187751936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202801.668862:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202801.668871:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202801/real 1788202801] req@ffff8e828793f800 x1875062187751936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202801.668880:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187751936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202801.668886:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202801.668888:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202801.668891:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187751936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202801.668896:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187752064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202801.668915:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202801.668927:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202801/real 1788202801] req@ffff8e828793e680 x1875062187752064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202801.668941:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187752064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202801.668955:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202801.668958:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202801.668961:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187752064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202801.668965:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187752192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202801.668982:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202801.668990:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202801/real 1788202801] req@ffff8e828793f100 x1875062187752192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202801.668998:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187752192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202801.669004:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202801.669006:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202801.669008:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187752192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202801.669013:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187752320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202801.669031:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202801.669042:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202801/real 1788202801] req@ffff8e828793c700 x1875062187752320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202801.669056:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187752320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202801.669064:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202801.669067:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202801.669071:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187752320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202802.372525:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202803.396503:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202804.420502:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202805.444507:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202806.468538:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202806.788445:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202806.788457:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202806.788462:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202806.788479:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4261 00000100:00080000:3.0:1788202806.788491:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202806.788499:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202806.788545:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187752448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202806.788572:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202806.788576:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202806.788577:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202806.788581:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4261 00000100:00080000:3.0:1788202806.788584:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202806.788587:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202806.788597:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202806.788610:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187752576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202806.788618:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202806.788621:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202806.788624:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788202806.788625:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187752448:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202806.788628:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4261 00000100:00080000:3.0:1788202806.788634:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202806.788637:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202806.788641:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202806.788655:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187752704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202806.788661:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202806.788664:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202806.788666:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202806.788669:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4261 00000100:00080000:3.0:1788202806.788672:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202806.788675:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202806.788678:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202806.788686:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187752832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202806.788697:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202806.788700:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202806.788702:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202806.788704:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4261 00000100:00080000:3.0:1788202806.788706:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202806.788708:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202806.788711:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788202806.788717:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202806.788720:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187752960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202806.788773:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202806/real 1788202806] req@ffff8e8289538a80 x1875062187752448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202806.788789:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187752448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202806.788800:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202806.788803:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202806.788807:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187752448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202806.788829:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187752576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202806.788845:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202806.788854:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202806/real 1788202806] req@ffff8e828953a680 x1875062187752576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202806.788862:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187752576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202806.788868:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202806.788870:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202806.788872:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187752576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202806.788877:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187752704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202806.788888:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202806.788894:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202806/real 1788202806] req@ffff8e828953b100 x1875062187752704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202806.788902:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187752704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202806.788907:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202806.788908:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202806.788910:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187752704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202806.788913:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187752832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202806.788924:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202806.788929:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202806/real 1788202806] req@ffff8e8289539500 x1875062187752832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202806.788936:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187752832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202806.788941:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202806.788942:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202806.788944:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187752832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202806.788947:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187752960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202806.788956:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202806.788962:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202806/real 1788202806] req@ffff8e8289539880 x1875062187752960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202806.788970:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187752960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202806.788975:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202806.788976:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202806.788978:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187752960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788202807.492506:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202808.516539:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202809.540520:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202810.564485:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202811.588501:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202811.908469:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202811.908479:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202811.908484:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202811.908495:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4266 00000100:00080000:0.0:1788202811.908507:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202811.908515:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202811.908562:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187753088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202811.908589:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202811.908592:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202811.908593:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202811.908597:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4266 00000100:00080000:0.0:1788202811.908599:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202811.908602:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202811.908611:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202811.908621:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187753216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202811.908641:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187753088:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202811.908648:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202811.908653:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202811.908656:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202811.908661:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4266 00000100:00080000:0.0:1788202811.908666:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202811.908671:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202811.908677:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202811.908693:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187753344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202811.908703:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202811.908707:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202811.908709:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202811.908713:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4266 00000100:00080000:0.0:1788202811.908719:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202811.908723:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202811.908727:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788202811.908736:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202811.908738:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187753472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202811.908747:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202811.908750:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202811.908751:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202811.908754:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4266 00000100:00080000:0.0:1788202811.908757:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202811.908759:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202811.908761:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202811.908767:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187753600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202811.908817:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202811/real 1788202811] req@ffff8e828793e300 x1875062187753088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202811.908841:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187753088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202811.908855:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202811.908860:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202811.908865:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187753088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202811.908896:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187753216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202811.908919:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202811.908930:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202811/real 1788202811] req@ffff8e828793c000 x1875062187753216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202811.908939:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187753216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202811.908945:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202811.908947:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202811.908950:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187753216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202811.908954:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187753344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202811.908966:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202811.908972:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202811/real 1788202811] req@ffff8e828793dc00 x1875062187753344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202811.908980:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187753344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202811.908985:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202811.908987:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202811.908989:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187753344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202811.908994:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187753472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202811.909006:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202811.909012:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202811/real 1788202811] req@ffff8e828793f480 x1875062187753472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202811.909020:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187753472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202811.909025:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202811.909027:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202811.909029:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187753472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202811.909032:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187753600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202811.909042:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202811.909048:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202811/real 1788202811] req@ffff8e828793ca80 x1875062187753600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202811.909055:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187753600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202811.909060:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202811.909062:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202811.909063:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187753600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202812.612545:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202813.636539:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202814.660726:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202815.684455:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202816.708490:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202817.028454:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202817.028462:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202817.028466:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202817.028480:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4272 00000100:00080000:3.0:1788202817.028490:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202817.028495:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202817.028528:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187753728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202817.028549:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202817.028551:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202817.028553:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202817.028556:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4272 00000100:00080000:3.0:1788202817.028558:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202817.028560:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202817.028568:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202817.028584:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187753856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202817.028592:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202817.028594:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202817.028595:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202817.028598:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4272 00000100:00080000:3.0:1788202817.028601:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202817.028603:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202817.028606:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202817.028613:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187753728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202817.028615:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187753984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202817.028636:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202817.028640:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202817.028642:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202817.028646:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4272 00000100:00080000:3.0:1788202817.028650:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202817.028654:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202817.028660:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202817.028674:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187754112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202817.028684:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202817.028688:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202817.028690:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788202817.028694:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788202817.028698:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4272 00000100:00080000:3.0:1788202817.028702:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202817.028705:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202817.028709:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202817.028718:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187754240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202817.028770:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202817/real 1788202817] req@ffff8e828953bb80 x1875062187753728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202817.028794:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187753728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202817.028806:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202817.028810:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202817.028815:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187753728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202817.028822:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187753856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202817.028845:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202817.028858:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202817/real 1788202817] req@ffff8e8289538380 x1875062187753856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202817.028871:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187753856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202817.028880:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202817.028883:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202817.028887:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187753856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202817.028909:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187753984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202817.028928:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202817.028940:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202817/real 1788202817] req@ffff8e8289539f80 x1875062187753984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202817.028959:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187753984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202817.028968:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202817.028971:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202817.028975:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187753984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202817.028981:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187754112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202817.029023:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202817.029036:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202817/real 1788202817] req@ffff8e8289539880 x1875062187754112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202817.029048:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187754112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202817.029057:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202817.029060:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202817.029063:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187754112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202817.029069:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187754240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202817.029089:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202817.029099:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202817/real 1788202817] req@ffff8e8289539500 x1875062187754240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202817.029111:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187754240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202842 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202817.029125:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202817.029128:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202817.029132:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187754240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202817.732495:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202818.756540:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202819.780629:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202820.804591:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202821.828463:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202822.148442:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202822.148454:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202822.148459:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202822.148479:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4277 00000100:00080000:3.0:1788202822.148490:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202822.148502:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202822.148549:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187754368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202822.148582:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202822.148586:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202822.148589:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202822.148594:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4277 00000100:00080000:3.0:1788202822.148598:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202822.148603:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202822.148613:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202822.148637:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187754368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202822.148639:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187754496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202822.148663:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202822.148668:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202822.148671:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202822.148677:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4277 00000100:00080000:3.0:1788202822.148683:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202822.148689:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202822.148696:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202822.148710:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187754624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202822.148721:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788202822.148726:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788202822.148728:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202822.148732:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202822.148737:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4277 00000100:00080000:3.0:1788202822.148742:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202822.148747:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202822.148752:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202822.148766:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187754752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202822.148777:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202822/real 1788202822] req@ffff8e828953b100 x1875062187754368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202822.148792:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187754368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202822.148802:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202822.148805:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202822.148809:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187754368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202822.148830:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187754496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202822.148857:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202822.148869:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202822/real 1788202822] req@ffff8e828953a680 x1875062187754496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202822.148879:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187754496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202822.148893:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202822.148895:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202822.148915:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187754496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202822.148921:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187754624:192.168.201.121@tcp:38:kworker.0 00000100:00080000:3.0:1788202822.148933:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202822.148937:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1788202822.148939:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788202822.148943:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202822.148954:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4277 00000100:00100000:2.0:1788202822.148961:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202822/real 1788202822] req@ffff8e8289538a80 x1875062187754624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202822.148961:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202822.148973:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202822.148978:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187754624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788202822.148982:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788202822.148986:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202822.148989:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202822.148992:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187754624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202822.149004:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187754752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202822.149026:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202822.149034:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202822/real 1788202822] req@ffff8e8289539500 x1875062187754752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202822.149044:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187754752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202822.149053:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202822.149056:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202822.149059:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187754752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788202822.149152:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187754880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202822.149232:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187754880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202822.149301:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202822.149362:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202822/real 1788202822] req@ffff8e8289539880 x1875062187754880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202822.149390:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187754880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202847 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202822.149404:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202822.149410:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202822.149415:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187754880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788202822.852487:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202823.876483:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202824.900537:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202825.924477:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202826.948564:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202827.268478:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202827.268493:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202827.268499:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202827.268525:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4282 00000100:00080000:2.0:1788202827.268541:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202827.268557:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202827.268616:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187755008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202827.268651:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202827.268655:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202827.268658:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202827.268664:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4282 00000100:00080000:2.0:1788202827.268668:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202827.268672:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202827.268682:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202827.268698:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187755136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202827.268700:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187755008:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202827.268715:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202827.268719:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202827.268720:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202827.268723:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4282 00000100:00080000:2.0:1788202827.268726:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202827.268728:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202827.268732:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202827.268740:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187755264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202827.268747:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202827.268750:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202827.268751:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202827.268754:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4282 00000100:00080000:2.0:1788202827.268757:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202827.268759:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202827.268762:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202827.268769:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187755392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202827.268775:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202827.268777:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202827.268778:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202827.268785:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4282 00000100:00080000:2.0:1788202827.268787:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788202827.268790:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788202827.268790:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202827.268793:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202827.268801:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187755520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202827.268871:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202827/real 1788202827] req@ffff8e82894e6680 x1875062187755008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202827.268890:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187755008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202827.268900:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202827.268904:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202827.268908:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187755008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202827.268936:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187755136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202827.268955:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202827.268969:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202827/real 1788202827] req@ffff8e82894e7480 x1875062187755136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202827.268982:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187755136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202827.268993:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202827.268996:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202827.269000:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187755136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202827.269006:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187755264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202827.269029:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202827.269041:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202827/real 1788202827] req@ffff8e82894e6a00 x1875062187755264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202827.269054:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187755264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202827.269061:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202827.269064:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202827.269067:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187755264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202827.269074:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187755392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202827.269096:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202827.269108:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202827/real 1788202827] req@ffff8e82894e4000 x1875062187755392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202827.269121:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187755392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202827.269130:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202827.269133:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202827.269137:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187755392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202827.269143:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187755520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202827.269165:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202827.269177:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202827/real 1788202827] req@ffff8e82894e5880 x1875062187755520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202827.269194:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187755520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202852 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202827.269200:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202827.269202:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202827.269204:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187755520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202827.972452:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202828.996440:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202830.020511:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202831.044454:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202832.068551:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202832.388417:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202832.388434:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202832.388440:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202832.388454:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4287 00000100:00080000:3.0:1788202832.388469:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202832.388479:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202832.388534:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187755648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202832.388571:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202832.388577:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202832.388579:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202832.388585:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4287 00000100:00080000:3.0:1788202832.388591:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202832.388597:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202832.388611:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202832.388638:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187755648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202832.388638:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187755776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202832.388681:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202832.388690:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202832.388692:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202832.388698:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4287 00000100:00080000:3.0:1788202832.388705:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202832.388710:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202832.388718:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788202832.388741:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202832.388751:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187755904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202832.388767:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202832.388771:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202832.388774:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202832.388779:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4287 00000100:00080000:3.0:1788202832.388785:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202832.388790:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202832.388797:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202832.388800:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202832/real 1788202832] req@ffff8e8289539f80 x1875062187755648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202832.388820:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187756032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202832.388827:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187755648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202832.388859:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202832.388863:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788202832.388865:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202832.388872:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202832.388874:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:3.0:1788202832.388889:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4287 00000100:00100000:1.0:1788202832.388890:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187755648:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202832.388901:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202832.388910:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202832.388917:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202832.388928:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187755776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788202832.388939:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187756160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202832.388965:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202832.388984:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202832/real 1788202832] req@ffff8e8289538380 x1875062187755776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202832.389005:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187755776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202832.389014:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202832.389017:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202832.389021:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187755776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202832.389028:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187755904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202832.389047:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202832.389057:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202832/real 1788202832] req@ffff8e828953bb80 x1875062187755904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202832.389069:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187755904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202832.389077:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202832.389080:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202832.389083:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187755904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202832.389090:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187756032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202832.389107:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202832.389117:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202832/real 1788202832] req@ffff8e8289539880 x1875062187756032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202832.389128:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187756032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202832.389143:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202832.389146:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202832.389150:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187756032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202832.389165:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187756160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202832.389183:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202832.389193:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202832/real 1788202832] req@ffff8e8289539500 x1875062187756160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202832.389204:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187756160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202832.389212:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202832.389215:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202832.389219:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187756160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202833.092463:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202834.116510:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202835.140461:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202836.164522:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202837.188535:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202837.508465:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202837.508474:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202837.508478:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202837.508486:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4292 00000100:00080000:2.0:1788202837.508495:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202837.508501:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202837.508534:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187756288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202837.508558:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202837.508561:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202837.508563:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202837.508566:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4292 00000100:00080000:2.0:1788202837.508568:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202837.508570:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202837.508578:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202837.508588:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187756416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202837.508595:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202837.508598:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202837.508599:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202837.508601:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4292 00000100:00080000:2.0:1788202837.508604:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202837.508608:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202837.508609:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187756288:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788202837.508612:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202837.508621:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187756544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202837.508629:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202837.508647:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202837.508650:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202837.508654:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4292 00000100:00080000:2.0:1788202837.508657:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202837.508661:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202837.508668:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202837.508682:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187756672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202837.508692:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202837.508696:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202837.508698:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202837.508702:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4292 00000100:00080000:2.0:1788202837.508706:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202837.508711:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202837.508715:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202837.508726:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187756800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202837.508737:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187756416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202837.508783:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187756544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202837.508807:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187756672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202837.508838:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187756800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788202838.212459:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202839.236542:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202840.260572:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788202840.580712:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788202840.580768:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788202840.580787:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202840.580841:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202840.580853:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202840.580862:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202840.580871:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788202840.581007:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202837/real 1788202840] req@ffff8e82894e6a00 x1875062187756544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202840.581038:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187756544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202840.581053:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202840.581058:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202840.581073:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187756544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202840.581083:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202837/real 1788202840] req@ffff8e82894e7480 x1875062187756672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202840.581097:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187756672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202840.581107:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202840.581111:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202840.581115:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187756672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202840.581123:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202837/real 1788202840] req@ffff8e82894e6680 x1875062187756800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202840.581135:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187756800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202840.581150:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202840.581154:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202840.581157:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187756800:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202840.581165:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202837/real 1788202840] req@ffff8e82894e5880 x1875062187756288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202840.581178:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187756288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202840.581187:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202840.581190:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202840.581194:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187756288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202840.581201:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202837/real 1788202840] req@ffff8e82894e4000 x1875062187756416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202840.581214:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187756416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202840.581222:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202840.581226:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202840.581229:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187756416:192.168.201.121@tcp:38:kworker.0 00010000:00010000:2.0:1788202841.284514:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202842.308528:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202842.628455:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202842.628470:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202842.628476:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202842.628491:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4297 00000100:00080000:2.0:1788202842.628505:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202842.628514:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202842.628597:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187756928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202842.628639:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202842.628644:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202842.628646:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202842.628651:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4297 00000100:00080000:2.0:1788202842.628655:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202842.628660:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202842.628673:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202842.628704:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187756928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202842.628707:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187757056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202842.628726:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202842.628729:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202842.628732:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202842.628741:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4297 00000100:00080000:2.0:1788202842.628746:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202842.628750:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202842.628758:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202842.628772:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187757184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202842.628805:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788202842.628808:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788202842.628811:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202842.628815:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202842.628822:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4297 00000100:00080000:2.0:1788202842.628827:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202842.628833:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202842.628840:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202842.628854:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187757312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202842.628865:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202842.628869:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202842.628871:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202842.628876:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4297 00000100:00080000:2.0:1788202842.628880:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202842.628885:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202842.628890:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202842.628893:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202842/real 1788202842] req@ffff8e82894e7100 x1875062187756928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202842.628906:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187757440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202842.628925:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187756928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202842.628941:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202842.628946:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202842.628950:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187756928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202842.628987:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187757056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202842.629015:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202842.629031:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202842/real 1788202842] req@ffff8e82894e4a80 x1875062187757056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202842.629044:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187757056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202842.629053:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202842.629056:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202842.629060:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187757056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202842.629066:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187757184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202842.629089:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202842.629102:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202842/real 1788202842] req@ffff8e82894e7800 x1875062187757184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202842.629114:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187757184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202842.629128:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202842.629132:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202842.629135:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187757184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202842.629141:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187757312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202842.629163:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202842.629174:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202842/real 1788202842] req@ffff8e82894e6300 x1875062187757312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202842.629186:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187757312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202842.629193:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202842.629196:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202842.629200:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187757312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202842.629206:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187757440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202842.629227:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202842.629238:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202842/real 1788202842] req@ffff8e82894e4e00 x1875062187757440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202842.629250:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187757440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202842.629258:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202842.629261:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202842.629266:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187757440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202843.332507:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202844.356441:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202845.380492:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202846.404517:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202847.428441:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202847.748488:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202847.748499:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202847.748503:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202847.748512:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4302 00000100:00080000:3.0:1788202847.748520:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202847.748526:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202847.748565:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187757568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202847.748600:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202847.748603:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202847.748604:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202847.748608:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4302 00000100:00080000:3.0:1788202847.748611:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202847.748614:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202847.748623:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202847.748635:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187757696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202847.748643:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202847.748645:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202847.748646:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202847.748650:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4302 00000100:00080000:3.0:1788202847.748652:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202847.748655:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202847.748662:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202847.748673:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187757568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202847.748678:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187757824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202847.748696:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202847.748698:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202847.748699:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202847.748703:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4302 00000100:00080000:3.0:1788202847.748705:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202847.748708:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202847.748712:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202847.748721:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187757952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202847.748728:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202847.748730:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202847.748732:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202847.748736:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4302 00000100:00080000:3.0:1788202847.748738:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202847.748742:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202847.748746:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202847.748752:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187758080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202847.748771:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202847.748821:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202847/real 1788202847] req@ffff8e8289538a80 x1875062187757568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202847.748837:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187757568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202847.748847:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202847.748851:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202847.748854:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187757568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202847.748860:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187757696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202847.748877:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202847.748890:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202847/real 1788202847] req@ffff8e828953a680 x1875062187757696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202847.748899:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187757696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202847.748905:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202847.748907:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202847.748910:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187757696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202847.748936:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187757824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202847.748949:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202847.748956:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202847/real 1788202847] req@ffff8e828953b100 x1875062187757824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202847.748964:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187757824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202847.748969:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202847.748971:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202847.748973:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187757824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202847.748977:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187757952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202847.748989:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202847.748995:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202847/real 1788202847] req@ffff8e8289539500 x1875062187757952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202847.749003:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187757952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202847.749008:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202847.749010:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202847.749011:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187757952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202847.749015:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187758080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202847.749025:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202847.749030:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202847/real 1788202847] req@ffff8e8289539880 x1875062187758080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202847.749039:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187758080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202847.749044:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202847.749046:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202847.749049:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187758080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202848.452479:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202849.476610:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202850.500469:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202851.524510:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202852.548519:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202852.868426:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202852.868435:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202852.868439:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202852.868448:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4307 00000100:00080000:3.0:1788202852.868464:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202852.868471:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202852.868504:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187758208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202852.868527:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202852.868530:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202852.868532:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202852.868535:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4307 00000100:00080000:3.0:1788202852.868537:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202852.868540:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202852.868553:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202852.868564:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187758336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202852.868575:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187758208:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202852.868581:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202852.868584:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202852.868585:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202852.868589:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4307 00000100:00080000:3.0:1788202852.868591:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202852.868594:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202852.868597:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202852.868604:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187758464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202852.868611:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202852.868613:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202852.868615:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202852.868617:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4307 00000100:00080000:3.0:1788202852.868620:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202852.868623:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202852.868626:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202852.868636:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187758592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202852.868637:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788202852.868646:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202852.868649:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202852.868650:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202852.868654:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4307 00000100:00080000:3.0:1788202852.868657:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202852.868659:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202852.868662:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202852.868669:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187758720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202852.868690:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202852/real 1788202852] req@ffff8e828953bb80 x1875062187758208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202852.868710:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187758208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202852.868721:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202852.868726:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202852.868729:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187758208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202852.868748:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187758336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202852.868774:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202852.868783:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202852/real 1788202852] req@ffff8e8289538380 x1875062187758336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202852.868797:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187758336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202852.868803:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202852.868805:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202852.868807:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187758336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202852.868811:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187758464:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202852.868822:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202852.868829:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202852/real 1788202852] req@ffff8e8289539f80 x1875062187758464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202852.868836:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187758464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202852.868841:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202852.868843:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202852.868851:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187758464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202852.868855:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187758592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202852.868870:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202852.868877:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202852/real 1788202852] req@ffff8e8289539880 x1875062187758592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202852.868884:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187758592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202852.868893:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202852.868895:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202852.868897:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187758592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202852.868900:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187758720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202852.868910:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202852.868916:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202852/real 1788202852] req@ffff8e8289539500 x1875062187758720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202852.868922:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187758720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202852.868926:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202852.868928:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202852.868930:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187758720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788202853.572473:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202854.596506:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202855.620609:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202856.644485:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202857.668534:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202857.988416:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202857.988425:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202857.988429:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202857.988437:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4313 00000100:00080000:3.0:1788202857.988447:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202857.988453:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202857.988486:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187758848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202857.988513:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202857.988516:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202857.988517:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202857.988520:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4313 00000100:00080000:3.0:1788202857.988523:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202857.988525:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202857.988533:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202857.988544:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187758976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202857.988552:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202857.988558:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202857.988559:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202857.988562:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4313 00000100:00080000:3.0:1788202857.988565:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202857.988570:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202857.988571:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187758848:192.168.201.121@tcp:250:kworker.0 00000008:00100000:3.0:1788202857.988573:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202857.988583:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187759104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202857.988590:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202857.988592:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202857.988593:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202857.988596:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4313 00000100:00080000:3.0:1788202857.988598:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202857.988600:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202857.988604:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202857.988611:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187759232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202857.988617:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202857.988619:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202857.988620:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202857.988622:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4313 00000100:00080000:3.0:1788202857.988625:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202857.988628:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202857.988633:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788202857.988636:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202857.988649:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187759360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202857.988687:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202857/real 1788202857] req@ffff8e828953b100 x1875062187758848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202857.988708:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187758848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202857.988727:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202857.988733:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202857.988747:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187758848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202857.988756:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187758976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202857.988788:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202857.988804:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202857/real 1788202857] req@ffff8e828953a680 x1875062187758976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202857.988818:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187758976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202857.988828:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202857.988832:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202857.988836:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187758976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202857.988867:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187759104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202857.988893:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202857.988914:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202857/real 1788202857] req@ffff8e8289538a80 x1875062187759104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202857.988925:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187759104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202857.988932:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202857.988935:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202857.988939:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187759104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202857.988947:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187759232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202857.988963:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202857.988987:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202857/real 1788202857] req@ffff8e8289539500 x1875062187759232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202857.989015:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187759232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202857.989020:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202857.989023:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202857.989027:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187759232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202857.989037:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187759360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202857.989063:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202857.989084:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202857/real 1788202857] req@ffff8e8289539880 x1875062187759360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202857.989093:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187759360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202857.989105:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202857.989108:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202857.989110:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187759360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202858.692484:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202859.716471:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202860.740481:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202861.764475:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202862.788467:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202863.108431:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202863.108443:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202863.108447:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202863.108469:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4318 00000100:00080000:3.0:1788202863.108481:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202863.108487:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202863.108521:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187759488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202863.108547:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202863.108550:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202863.108551:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202863.108554:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4318 00000100:00080000:3.0:1788202863.108557:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202863.108559:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202863.108567:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202863.108577:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187759616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202863.108585:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202863.108587:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202863.108588:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202863.108591:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4318 00000100:00080000:3.0:1788202863.108594:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202863.108597:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202863.108600:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202863.108606:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187759744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202863.108614:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187759488:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202863.108615:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202863.108621:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202863.108624:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202863.108629:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4318 00000100:00080000:3.0:1788202863.108631:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202863.108633:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202863.108637:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202863.108645:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187759872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202863.108651:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202863.108653:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202863.108655:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202863.108657:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4318 00000100:00080000:3.0:1788202863.108660:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202863.108664:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202863.108670:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788202863.108681:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187760000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202863.108708:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202863.108758:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202863/real 1788202863] req@ffff8e8289539f80 x1875062187759488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202863.108789:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187759488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202863.108802:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202863.108807:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202863.108813:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187759488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202863.108824:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187759616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202863.108849:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202863.108865:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202863/real 1788202863] req@ffff8e8289538380 x1875062187759616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202863.108881:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187759616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202863.108892:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202863.108896:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202863.108899:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187759616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202863.108935:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187759744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202863.108960:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202863.108974:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202863/real 1788202863] req@ffff8e828953bb80 x1875062187759744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202863.108994:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187759744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202863.109003:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202863.109006:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202863.109010:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187759744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202863.109016:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187759872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202863.109036:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202863.109047:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202863/real 1788202863] req@ffff8e8289539880 x1875062187759872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202863.109061:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187759872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202863.109070:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202863.109074:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202863.109078:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187759872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202863.109084:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187760000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202863.109103:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202863.109113:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202863/real 1788202863] req@ffff8e8289539500 x1875062187760000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202888 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202863.109125:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187760000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202888 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202863.109140:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202863.109144:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202863.109147:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187760000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202863.812490:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202864.836486:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202865.860438:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202866.884518:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202867.908517:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202868.228486:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202868.228501:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202868.228507:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202868.228522:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4323 00000100:00080000:2.0:1788202868.228537:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202868.228545:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202868.228595:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187760128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202868.228629:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202868.228633:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202868.228635:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202868.228645:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4323 00000100:00080000:2.0:1788202868.228649:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202868.228653:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202868.228665:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202868.228680:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187760256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202868.228692:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202868.228698:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788202868.228699:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187760128:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202868.228701:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202868.228712:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4323 00000100:00080000:2.0:1788202868.228716:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202868.228720:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202868.228726:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202868.228738:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187760384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202868.228748:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202868.228752:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202868.228754:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202868.228759:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4323 00000100:00080000:2.0:1788202868.228763:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202868.228767:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202868.228772:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202868.228783:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187760512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202868.228794:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788202868.228799:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788202868.228800:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202868.228804:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202868.228809:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4323 00000100:00080000:2.0:1788202868.228814:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202868.228819:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202868.228824:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202868.228840:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187760640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202868.228865:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202868/real 1788202868] req@ffff8e82894e6d80 x1875062187760128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202868.228889:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187760128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202868.228902:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202868.228907:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202868.228911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187760128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202868.228942:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187760256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202868.228970:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202868.228985:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202868/real 1788202868] req@ffff8e82894e5c00 x1875062187760256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202868.228999:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187760256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202868.229016:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202868.229020:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202868.229036:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187760256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202868.229043:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187760384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202868.229066:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202868.229079:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202868/real 1788202868] req@ffff8e82894e4380 x1875062187760384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202868.229092:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187760384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202868.229118:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202868.229122:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202868.229126:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187760384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202868.229133:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187760512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202868.229159:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202868.229172:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202868/real 1788202868] req@ffff8e82894e5f80 x1875062187760512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202868.229185:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187760512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202868.229193:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202868.229197:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202868.229201:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187760512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202868.229207:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187760640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202868.229243:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202868.229257:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202868/real 1788202868] req@ffff8e82894e4e00 x1875062187760640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202893 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202868.229269:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187760640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202893 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202868.229278:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202868.229281:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202868.229285:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187760640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788202868.932490:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202869.956479:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202870.980512:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202872.004452:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202873.028534:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202873.348468:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202873.348484:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202873.348489:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202873.348501:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4328 00000100:00080000:3.0:1788202873.348515:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202873.348523:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202873.348568:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187760768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202873.348599:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202873.348604:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202873.348610:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202873.348616:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4328 00000100:00080000:3.0:1788202873.348621:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202873.348628:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187760768:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202873.348629:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202873.348647:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202873.348666:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187760896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202873.348676:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202873.348679:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202873.348680:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202873.348683:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4328 00000100:00080000:3.0:1788202873.348688:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202873.348692:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202873.348700:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788202873.348706:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202873.348717:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187761024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202873.348730:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202873.348734:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202873.348736:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202873.348740:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4328 00000100:00080000:3.0:1788202873.348744:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202873.348749:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202873.348756:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202873.348768:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202873/real 1788202873] req@ffff8e8289538a80 x1875062187760768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202873.348773:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187761152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202873.348799:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202873.348803:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202873.348805:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788202873.348808:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187760768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788202873.348813:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4328 00000100:00080000:3.0:1788202873.348820:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202873.348827:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:1.0:1788202873.348829:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788202873.348832:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788202873.348838:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202873.348845:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187760768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788202873.348849:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187761280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202873.348893:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187760896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202873.348921:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202873.348936:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202873/real 1788202873] req@ffff8e828953a680 x1875062187760896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202873.348951:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187760896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202873.348960:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202873.348965:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202873.348968:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187760896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202873.348975:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187761024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202873.348999:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202873.349030:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202873/real 1788202873] req@ffff8e828953b100 x1875062187761024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202873.349044:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187761024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202873.349050:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202873.349053:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202873.349055:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187761024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202873.349059:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187761152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202873.349077:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202873.349084:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202873/real 1788202873] req@ffff8e8289539500 x1875062187761152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202873.349093:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187761152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202873.349098:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202873.349100:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202873.349102:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187761152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202873.349106:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187761280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202873.349117:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202873.349123:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202873/real 1788202873] req@ffff8e8289539880 x1875062187761280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202873.349130:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187761280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202873.349135:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202873.349138:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202873.349139:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187761280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202874.052483:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202875.076639:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202876.100510:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202877.124538:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202878.148452:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788202878.468434:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202878.468446:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788202878.468452:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202878.468464:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4333 00000100:00080000:3.0:1788202878.468475:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202878.468482:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788202878.468529:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187761408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202878.468558:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202878.468563:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788202878.468565:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202878.468569:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4333 00000100:00080000:3.0:1788202878.468573:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202878.468578:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202878.468589:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202878.468609:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187761536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202878.468626:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187761408:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788202878.468629:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202878.468638:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788202878.468641:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202878.468646:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4333 00000100:00080000:3.0:1788202878.468651:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202878.468656:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202878.468662:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788202878.468680:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187761664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202878.468698:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202878.468702:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788202878.468704:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202878.468710:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4333 00000100:00080000:3.0:1788202878.468715:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202878.468720:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202878.468726:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788202878.468727:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788202878.468743:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187761792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788202878.468751:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788202878.468754:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788202878.468755:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788202878.468767:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4333 00000100:00080000:3.0:1788202878.468770:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788202878.468773:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788202878.468780:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202878.468789:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202878/real 1788202878] req@ffff8e828953bb80 x1875062187761408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788202878.468790:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187761920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202878.468845:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187761408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202878.468858:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202878.468863:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202878.468868:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187761408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202878.468898:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187761536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202878.468961:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202878.468978:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202878/real 1788202878] req@ffff8e8289538380 x1875062187761536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202878.468992:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187761536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202878.469002:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202878.469006:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202878.469010:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187761536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202878.469017:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187761664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202878.469041:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202878.469055:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202878/real 1788202878] req@ffff8e8289539f80 x1875062187761664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202878.469068:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187761664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202878.469084:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202878.469088:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202878.469091:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187761664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202878.469098:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187761792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202878.469121:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202878.469133:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202878/real 1788202878] req@ffff8e8289539880 x1875062187761792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202878.469145:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187761792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202878.469154:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202878.469158:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202878.469161:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187761792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202878.469168:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187761920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202878.469189:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202878.469199:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202878/real 1788202878] req@ffff8e8289539500 x1875062187761920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202878.469210:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187761920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202878.469219:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202878.469222:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202878.469226:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187761920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202879.172529:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202880.196540:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202881.220503:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202882.244521:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202883.268566:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202883.588452:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202883.588468:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202883.588473:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202883.588487:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4338 00000100:00080000:2.0:1788202883.588501:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202883.588511:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202883.588571:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187762048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202883.588603:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202883.588607:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202883.588609:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202883.588614:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4338 00000100:00080000:2.0:1788202883.588618:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202883.588630:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202883.588642:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202883.588661:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187762048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202883.588661:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187762176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202883.588679:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202883.588683:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202883.588685:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202883.588689:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4338 00000100:00080000:2.0:1788202883.588694:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202883.588699:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202883.588704:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202883.588717:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187762304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202883.588727:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202883.588730:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202883.588732:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202883.588736:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4338 00000100:00080000:2.0:1788202883.588741:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202883.588745:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202883.588750:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788202883.588756:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202883.588766:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187762432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202883.588785:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202883.588789:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202883.588791:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202883.588795:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4338 00000100:00080000:2.0:1788202883.588799:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202883.588804:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202883.588809:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202883.588822:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187762560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202883.588829:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202883/real 1788202883] req@ffff8e82894e6300 x1875062187762048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202883.588885:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187762048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202883.588898:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202883.588903:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202883.588908:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187762048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202883.588937:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187762176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202883.588964:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202883.588979:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202883/real 1788202883] req@ffff8e82894e7800 x1875062187762176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202883.588994:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187762176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202883.589003:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202883.589007:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202883.589011:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187762176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202883.589019:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187762304:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202883.589041:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202883.589054:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202883/real 1788202883] req@ffff8e82894e4a80 x1875062187762304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202883.589068:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187762304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202883.589077:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202883.589080:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202883.589084:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187762304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202883.589090:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187762432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202883.589112:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202883.589125:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202883/real 1788202883] req@ffff8e82894e7100 x1875062187762432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202883.589138:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187762432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202883.589148:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202883.589151:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202883.589155:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187762432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202883.589161:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187762560:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202883.589182:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202883.589193:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202883/real 1788202883] req@ffff8e82894e4000 x1875062187762560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202883.589201:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187762560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202883.589207:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202883.589209:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202883.589211:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187762560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788202884.292508:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202885.316532:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202886.340529:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202887.364439:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202888.388542:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202888.708398:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202888.708406:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202888.708410:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202888.708420:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4343 00000100:00080000:0.0:1788202888.708430:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202888.708436:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202888.708472:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187762688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202888.708497:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202888.708500:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202888.708501:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202888.708504:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4343 00000100:00080000:0.0:1788202888.708507:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202888.708510:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202888.708518:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202888.708530:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187762816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202888.708531:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187762688:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202888.708546:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202888.708552:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202888.708554:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202888.708558:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4343 00000100:00080000:0.0:1788202888.708562:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202888.708566:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202888.708572:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202888.708585:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187762944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202888.708586:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202888.708598:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202888.708601:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202888.708603:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202888.708608:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4343 00000100:00080000:0.0:1788202888.708612:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202888.708616:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202888.708620:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202888.708623:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202888/real 1788202888] req@ffff8e828793fb80 x1875062187762688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202888.708631:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187763072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202888.708638:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187762688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202888.708641:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202888.708648:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202888.708650:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202888.708654:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202888.708655:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4343 00000100:00080000:0.0:1788202888.708661:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202888.708662:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:0.0:1788202888.708670:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202888.708670:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187762688:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788202888.708680:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202888.708692:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187762816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202888.708696:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187763200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202888.708710:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202888.708720:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202888/real 1788202888] req@ffff8e828793d500 x1875062187762816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202888.708736:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187762816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202888.708770:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202888.708774:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202888.708778:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187762816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202888.708784:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187762944:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202888.708801:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202888.708809:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202888/real 1788202888] req@ffff8e828793d880 x1875062187762944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202888.708816:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187762944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202888.708820:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202888.708822:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202888.708824:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187762944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202888.708827:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187763072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202888.708837:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202888.708842:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202888/real 1788202888] req@ffff8e828793c380 x1875062187763072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202888.708848:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187763072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202888.708856:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202888.708858:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202888.708860:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187763072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202888.708868:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187763200:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202888.708877:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202888.708881:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202888/real 1788202888] req@ffff8e828793fb80 x1875062187763200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202888.708887:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187763200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202888.708892:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202888.708893:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202888.708895:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187763200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202889.412474:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202890.436492:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202891.460476:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202892.484519:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202893.508448:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202893.828480:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202893.828486:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202893.828489:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202893.828497:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4348 00000100:00080000:0.0:1788202893.828506:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202893.828516:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202893.828551:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187763328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202893.828581:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202893.828584:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202893.828585:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202893.828589:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4348 00000100:00080000:0.0:1788202893.828592:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202893.828596:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202893.828604:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202893.828617:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187763456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202893.828624:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202893.828626:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202893.828627:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202893.828629:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4348 00000100:00080000:0.0:1788202893.828631:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202893.828634:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202893.828638:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202893.828644:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187763584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202893.828651:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202893.828664:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202893.828666:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202893.828668:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4348 00000100:00080000:0.0:1788202893.828671:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202893.828674:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202893.828678:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202893.828687:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187763712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202893.828696:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202893.828699:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202893.828701:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202893.828705:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4348 00000100:00080000:0.0:1788202893.828709:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202893.828714:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202893.828720:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202893.828738:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187763840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202893.828740:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187763328:192.168.201.121@tcp:250:kworker.0 00000800:00000100:1.0:1788202893.828846:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202893.828896:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202893/real 1788202893] req@ffff8e828793ca80 x1875062187763328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202893.828914:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187763328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202893.828929:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202893.828932:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202893.828936:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187763328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202893.828942:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187763456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202893.828963:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202893.828984:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202893/real 1788202893] req@ffff8e828793f480 x1875062187763456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202893.828998:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187763456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202893.829004:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202893.829007:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202893.829011:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187763456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202893.829016:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187763584:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202893.829042:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202893.829061:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202893/real 1788202893] req@ffff8e828793dc00 x1875062187763584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202893.829069:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187763584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202893.829075:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202893.829077:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202893.829080:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187763584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202893.829086:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187763712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202893.829099:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202893.829106:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202893/real 1788202893] req@ffff8e828793c000 x1875062187763712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202893.829113:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187763712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202893.829118:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202893.829120:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202893.829122:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187763712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202893.829145:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187763840:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202893.829156:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202893.829162:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202893/real 1788202893] req@ffff8e828793e300 x1875062187763840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202893.829170:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187763840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202893.829175:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202893.829176:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202893.829178:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187763840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202894.532511:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202895.556547:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202896.580498:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202897.604491:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202898.628497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202898.948448:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202898.948456:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202898.948459:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202898.948469:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4354 00000100:00080000:2.0:1788202898.948480:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202898.948488:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202898.948523:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187763968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202898.948546:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202898.948549:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202898.948550:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202898.948553:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4354 00000100:00080000:2.0:1788202898.948556:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202898.948558:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202898.948577:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202898.948587:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187764096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202898.948595:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202898.948599:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202898.948601:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202898.948604:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4354 00000100:00080000:2.0:1788202898.948608:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202898.948611:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187763968:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202898.948615:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202898.948622:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202898.948634:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187764224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202898.948642:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202898.948645:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202898.948646:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202898.948649:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4354 00000100:00080000:2.0:1788202898.948652:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202898.948654:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202898.948658:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202898.948665:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187764352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202898.948670:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202898.948672:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202898.948674:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788202898.948677:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788202898.948680:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4354 00000100:00080000:2.0:1788202898.948683:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202898.948685:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202898.948687:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202898.948695:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187764480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202898.948719:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202898/real 1788202898] req@ffff8e82894e5880 x1875062187763968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202898.948731:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187763968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202898.948739:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202898.948742:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202898.948746:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187763968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202898.948754:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187764096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202898.948773:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202898.948785:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202898/real 1788202898] req@ffff8e82894e6680 x1875062187764096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202898.948797:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187764096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202898.948804:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202898.948807:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202898.948811:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187764096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202898.948832:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187764224:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202898.948852:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202898.948863:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202898/real 1788202898] req@ffff8e82894e7480 x1875062187764224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202898.948880:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187764224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202898.948888:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202898.948891:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202898.948894:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187764224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202898.948900:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187764352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202898.948919:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202898.948930:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202898/real 1788202898] req@ffff8e82894e6a00 x1875062187764352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202898.948941:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187764352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202898.948949:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202898.948952:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202898.948955:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187764352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202898.948960:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187764480:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202898.948984:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202898.948996:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202898/real 1788202898] req@ffff8e82894e4000 x1875062187764480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202898.949008:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187764480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202898.949020:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202898.949022:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202898.949025:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187764480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788202899.652501:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202900.676465:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202901.700551:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202902.724484:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202903.748489:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202904.068382:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202904.068391:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202904.068396:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202904.068404:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4359 00000100:00080000:2.0:1788202904.068414:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202904.068420:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202904.068451:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187764608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202904.068490:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202904.068494:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202904.068496:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202904.068500:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4359 00000100:00080000:2.0:1788202904.068503:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202904.068507:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202904.068516:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202904.068528:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187764736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202904.068536:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187764608:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788202904.068540:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202904.068545:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202904.068547:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202904.068552:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4359 00000100:00080000:2.0:1788202904.068556:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202904.068559:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202904.068564:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202904.068573:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187764864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202904.068580:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202904.068584:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202904.068585:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202904.068588:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4359 00000100:00080000:2.0:1788202904.068592:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202904.068595:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202904.068600:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202904.068618:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187764992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202904.068626:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202904.068629:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202904.068631:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202904.068635:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4359 00000100:00080000:2.0:1788202904.068639:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202904.068642:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187764736:192.168.201.121@tcp:38:kworker.0 00000100:00080000:2.0:1788202904.068647:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202904.068654:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202904.068667:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187765120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202904.068668:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187764864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202904.068698:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187764992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202904.068712:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187765120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202904.772447:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202905.796461:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202906.820490:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788202907.140738:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788202907.140808:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788202907.140831:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202907.140902:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202907.140917:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202907.140928:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202907.140939:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788202907.141063:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202904/real 1788202907] req@ffff8e82894e4e00 x1875062187765120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202907.141097:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187765120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202907.141112:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202907.141119:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202907.141124:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187765120:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202907.141137:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202904/real 1788202907] req@ffff8e82894e4a80 x1875062187764736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202907.141153:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187764736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202907.141163:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202907.141166:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202907.141170:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187764736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202907.141178:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202904/real 1788202907] req@ffff8e82894e7800 x1875062187764864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202907.141190:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187764864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202907.141199:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202907.141203:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202907.141207:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187764864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202907.141222:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202904/real 1788202907] req@ffff8e82894e6300 x1875062187764992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202907.141237:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187764992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202907.141245:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202907.141249:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202907.141252:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187764992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202907.141259:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202904/real 1788202907] req@ffff8e82894e7100 x1875062187764608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202907.141273:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187764608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202929 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202907.141282:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202907.141286:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202907.141290:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187764608:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788202907.844556:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202908.868403:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202909.188417:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202909.188426:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202909.188430:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202909.188439:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4364 00000100:00080000:0.0:1788202909.188448:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202909.188454:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202909.188482:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187765248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202909.188515:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202909.188519:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202909.188520:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202909.188524:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4364 00000100:00080000:0.0:1788202909.188527:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202909.188531:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202909.188539:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202909.188550:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187765376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202909.188560:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202909.188563:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202909.188566:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202909.188570:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4364 00000100:00100000:1.0:1788202909.188572:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187765248:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202909.188574:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202909.188580:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202909.188585:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202909.188596:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187765504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202909.188604:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202909.188607:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202909.188609:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202909.188612:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4364 00000100:00080000:0.0:1788202909.188626:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202909.188631:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:1.0:1788202909.188636:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788202909.188637:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202909.188651:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187765632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202909.188660:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202909.188663:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202909.188665:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202909.188669:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4364 00000100:00080000:0.0:1788202909.188672:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202909.188677:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202909.188681:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202909/real 1788202909] req@ffff8e828793c700 x1875062187765248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788202909.188683:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202909.188697:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187765760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202909.188705:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187765248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202909.188719:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202909.188724:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202909.188729:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187765248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202909.188753:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187765376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202909.188773:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202909.188785:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202909/real 1788202909] req@ffff8e828793f100 x1875062187765376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202909.188797:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187765376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202909.188805:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202909.188808:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202909.188811:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187765376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202909.188817:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187765504:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202909.188834:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202909.188844:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202909/real 1788202909] req@ffff8e828793e680 x1875062187765504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202909.188854:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187765504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202909.188861:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202909.188864:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202909.188867:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187765504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202909.188873:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187765632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202909.188890:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202909.188900:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202909/real 1788202909] req@ffff8e828793f800 x1875062187765632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202909.188910:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187765632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202909.188917:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202909.188920:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202909.188923:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187765632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202909.188928:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187765760:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202909.188945:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202909.188954:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202909/real 1788202909] req@ffff8e828793ce00 x1875062187765760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202909.188964:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187765760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202934 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202909.188971:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202909.188974:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202909.188977:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187765760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202909.892519:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202910.916579:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202911.940471:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202912.964505:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202913.988493:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202914.308436:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202914.308454:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202914.308459:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202914.308470:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4369 00000100:00080000:0.0:1788202914.308482:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202914.308489:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202914.308534:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187765888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202914.308566:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202914.308569:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202914.308570:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202914.308574:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4369 00000100:00080000:0.0:1788202914.308577:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202914.308580:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202914.308591:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202914.308605:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187766016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202914.308617:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202914.308621:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202914.308624:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202914.308629:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4369 00000100:00080000:0.0:1788202914.308634:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788202914.308639:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187765888:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202914.308642:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202914.308650:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202914.308664:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187766144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202914.308672:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202914.308675:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202914.308676:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202914.308679:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4369 00000100:00080000:0.0:1788202914.308699:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202914.308702:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202914.308705:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788202914.308711:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202914.308715:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187766272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202914.308723:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202914.308725:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202914.308727:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202914.308730:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4369 00000100:00080000:0.0:1788202914.308736:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202914.308738:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202914.308741:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202914.308748:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187766400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202914.308761:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202914/real 1788202914] req@ffff8e828793ce00 x1875062187765888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202914.308777:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187765888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202914.308786:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202914.308789:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202914.308793:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187765888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202914.308798:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187766016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202914.308814:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202914.308822:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202914/real 1788202914] req@ffff8e828793f800 x1875062187766016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202914.308831:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187766016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202914.308837:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202914.308839:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202914.308841:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187766016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202914.308866:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187766144:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202914.308887:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202914.308898:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202914/real 1788202914] req@ffff8e828793e680 x1875062187766144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202914.308911:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187766144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202914.308920:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202914.308924:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202914.308927:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187766144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202914.308934:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187766272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202914.308956:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202914.308968:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202914/real 1788202914] req@ffff8e828793f100 x1875062187766272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202914.308984:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187766272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202914.308990:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202914.308992:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202914.308994:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187766272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202914.308998:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187766400:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202914.309013:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202914.309020:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202914/real 1788202914] req@ffff8e828793c700 x1875062187766400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202914.309027:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187766400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202914.309032:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202914.309034:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202914.309036:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187766400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788202915.012394:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202916.036447:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202917.060480:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202918.084480:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202919.108720:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202919.428465:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202919.428480:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202919.428488:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202919.428502:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4374 00000100:00080000:0.0:1788202919.428518:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202919.428529:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202919.428580:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187766528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202919.428618:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202919.428621:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202919.428623:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202919.428626:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4374 00000100:00080000:0.0:1788202919.428630:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202919.428634:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202919.428643:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202919.428666:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187766656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202919.428672:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187766528:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202919.428685:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202919.428691:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202919.428693:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202919.428698:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4374 00000100:00080000:0.0:1788202919.428703:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202919.428707:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202919.428712:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202919.428723:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187766784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202919.428735:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202919.428738:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202919.428739:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202919.428743:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4374 00000100:00080000:0.0:1788202919.428746:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202919.428749:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202919.428752:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202919.428763:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187766912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788202919.428765:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202919.428780:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202919.428784:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202919.428786:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202919.428792:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4374 00000100:00080000:0.0:1788202919.428796:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202919.428801:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202919.428806:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202919.428820:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187767040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202919.428826:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202919/real 1788202919] req@ffff8e828793e300 x1875062187766528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202919.428853:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187766528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202919.428868:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202919.428873:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202919.428878:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187766528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202919.428913:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187766656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202919.428937:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202919.428951:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202919/real 1788202919] req@ffff8e828793c000 x1875062187766656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202919.428965:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187766656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202919.428981:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202919.428986:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202919.428988:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187766656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202919.428994:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187766784:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202919.429010:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202919.429019:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202919/real 1788202919] req@ffff8e828793dc00 x1875062187766784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202919.429027:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187766784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202919.429033:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202919.429035:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202919.429037:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187766784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202919.429042:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187766912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202919.429054:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202919.429060:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202919/real 1788202919] req@ffff8e828793f480 x1875062187766912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202919.429068:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187766912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202919.429073:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202919.429075:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202919.429076:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187766912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202919.429080:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187767040:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202919.429090:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202919.429095:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202919/real 1788202919] req@ffff8e828793ca80 x1875062187767040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202919.429102:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187767040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202919.429107:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202919.429109:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202919.429111:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187767040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202920.132479:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202921.156394:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202922.180534:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202923.204509:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202924.228473:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202924.548448:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202924.548458:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202924.548463:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202924.548475:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4379 00000100:00080000:0.0:1788202924.548487:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202924.548494:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202924.548538:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187767168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202924.548576:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202924.548580:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202924.548585:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202924.548590:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4379 00000100:00080000:0.0:1788202924.548593:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202924.548598:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202924.548609:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202924.548623:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187767296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202924.548634:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187767168:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202924.548636:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202924.548644:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202924.548646:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202924.548650:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4379 00000100:00080000:0.0:1788202924.548653:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202924.548656:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202924.548661:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202924.548671:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187767424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202924.548685:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202924.548688:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202924.548690:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202924.548694:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4379 00000100:00080000:0.0:1788202924.548706:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202924.548710:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202924.548716:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788202924.548721:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202924.548730:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187767552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202924.548737:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202924.548740:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202924.548742:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202924.548745:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4379 00000100:00080000:0.0:1788202924.548750:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202924.548752:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202924.548755:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202924.548764:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187767680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788202924.548779:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202924/real 1788202924] req@ffff8e828793fb80 x1875062187767168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202924.548798:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187767168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202924.548807:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202924.548810:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202924.548813:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187767168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202924.548837:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187767296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202924.548856:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202924.548864:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202924/real 1788202924] req@ffff8e828793c380 x1875062187767296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202924.548872:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187767296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202924.548877:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202924.548879:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202924.548881:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187767296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202924.548884:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187767424:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788202924.548894:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202924.548913:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202924/real 1788202924] req@ffff8e828793d880 x1875062187767424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202924.548920:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187767424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202924.548929:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202924.548932:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202924.548934:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187767424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202924.548937:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187767552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202924.548948:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202924.548954:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202924/real 1788202924] req@ffff8e828793d500 x1875062187767552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202924.548962:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187767552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202924.548969:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202924.548973:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202924.548976:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187767552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788202924.548981:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187767680:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788202924.549000:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788202924.549010:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202924/real 1788202924] req@ffff8e828793ca80 x1875062187767680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202924.549021:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187767680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202924.549035:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202924.549039:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788202924.549043:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187767680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202925.252531:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202926.276487:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202927.300497:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202928.324427:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202929.348497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202929.668452:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202929.668463:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202929.668468:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202929.668478:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4384 00000100:00080000:0.0:1788202929.668488:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202929.668499:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202929.668544:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187767808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202929.668582:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202929.668586:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202929.668589:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202929.668596:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4384 00000100:00080000:0.0:1788202929.668600:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202929.668607:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202929.668618:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202929.668632:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187767808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202929.668640:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187767936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202929.668655:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202929.668665:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202929.668668:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202929.668673:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4384 00000100:00080000:0.0:1788202929.668680:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202929.668688:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202929.668694:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202929.668713:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187768064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202929.668732:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202929.668733:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202929.668739:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202929.668742:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202929.668748:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4384 00000100:00080000:0.0:1788202929.668756:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202929.668760:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202929.668767:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202929.668782:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202929/real 1788202929] req@ffff8e828793f480 x1875062187767808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202929.668787:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187768192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202929.668821:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202929.668828:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788202929.668828:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187767808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202929.668835:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202929.668855:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4384 00000100:00080000:2.0:1788202929.668858:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202929.668861:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202929.668864:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202929.668871:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187767808:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202929.668875:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202929.668885:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202929.668901:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187767936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202929.668903:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187768320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202929.668926:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202929.668938:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202929/real 1788202929] req@ffff8e828793dc00 x1875062187767936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202929.668948:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187767936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202929.668959:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202929.668962:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202929.668965:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187767936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202929.668972:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187768064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202929.668996:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202929.669009:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202929/real 1788202929] req@ffff8e828793c000 x1875062187768064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202929.669020:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187768064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202929.669027:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202929.669042:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202929.669046:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187768064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202929.669052:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187768192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202929.669073:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202929.669085:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202929/real 1788202929] req@ffff8e828793e300 x1875062187768192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202929.669097:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187768192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202929.669104:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202929.669106:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202929.669109:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187768192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202929.669129:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187768320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202929.669144:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202929.669151:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202929/real 1788202929] req@ffff8e828793c700 x1875062187768320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202929.669165:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187768320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202929.669172:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202929.669183:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202929.669187:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187768320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788202930.372458:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202931.396504:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202932.420555:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202933.444466:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202934.468546:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202934.788433:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202934.788444:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202934.788449:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202934.788460:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4389 00000100:00080000:0.0:1788202934.788472:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202934.788479:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202934.788519:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187768448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202934.788557:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202934.788562:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202934.788564:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202934.788568:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4389 00000100:00080000:0.0:1788202934.788572:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202934.788577:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202934.788588:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202934.788601:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187768576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202934.788610:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202934.788614:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202934.788616:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202934.788621:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4389 00000100:00080000:0.0:1788202934.788625:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202934.788629:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202934.788634:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202934.788644:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187768448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202934.788647:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187768704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202934.788657:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202934.788660:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202934.788661:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202934.788664:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4389 00000100:00080000:0.0:1788202934.788666:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202934.788669:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202934.788673:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202934.788689:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187768832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202934.788695:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202934.788697:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202934.788698:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202934.788701:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4389 00000100:00080000:0.0:1788202934.788713:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202934.788717:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:2.0:1788202934.788722:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788202934.788725:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202934.788749:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187768960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202934.788782:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202934/real 1788202934] req@ffff8e828793f100 x1875062187768448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202934.788802:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187768448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202934.788814:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202934.788817:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202934.788821:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187768448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202934.788827:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187768576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202934.788845:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202934.788859:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202934/real 1788202934] req@ffff8e828793e680 x1875062187768576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202934.788867:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187768576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202934.788872:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202934.788874:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202934.788876:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187768576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202934.788902:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187768704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202934.788920:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202934.788929:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202934/real 1788202934] req@ffff8e828793f800 x1875062187768704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202934.788940:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187768704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202934.788971:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202934.788974:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202934.788976:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187768704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202934.788980:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187768832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202934.788995:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202934.789002:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202934/real 1788202934] req@ffff8e828793ce00 x1875062187768832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202934.789014:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187768832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202934.789018:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202934.789020:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202934.789021:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187768832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202934.789024:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187768960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202934.789034:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202934.789039:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202934/real 1788202934] req@ffff8e828793c700 x1875062187768960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202934.789045:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187768960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202934.789049:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202934.789051:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202934.789052:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187768960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202935.492504:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202936.516467:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202937.540536:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202938.564558:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202939.588515:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202939.908472:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202939.908482:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202939.908487:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202939.908499:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4394 00000100:00080000:0.0:1788202939.908511:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202939.908519:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202939.908563:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187769088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202939.908592:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202939.908595:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202939.908597:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202939.908604:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4394 00000100:00080000:0.0:1788202939.908607:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202939.908610:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202939.908620:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202939.908632:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187769216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202939.908644:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202939.908648:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202939.908650:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788202939.908658:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187769088:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202939.908659:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4394 00000100:00080000:0.0:1788202939.908665:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202939.908672:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202939.908679:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202939.908693:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187769344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202939.908701:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202939.908704:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202939.908706:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202939.908710:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4394 00000100:00080000:0.0:1788202939.908713:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202939.908717:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202939.908721:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202939.908731:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187769472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202939.908737:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202939.908740:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202939.908741:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202939.908746:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4394 00000800:00000100:2.0:1788202939.908749:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202939.908750:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202939.908755:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202939.908761:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202939.908771:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187769600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202939.908824:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202939/real 1788202939] req@ffff8e828793e300 x1875062187769088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202939.908843:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187769088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202939.908876:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202939.908881:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202939.908887:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187769088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202939.908917:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187769216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202939.908944:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202939.908955:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202939/real 1788202939] req@ffff8e828793c000 x1875062187769216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202939.908965:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187769216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202939.908971:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202939.908974:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202939.908976:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187769216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202939.908980:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187769344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202939.908994:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202939.909001:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202939/real 1788202939] req@ffff8e828793dc00 x1875062187769344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202939.909008:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187769344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202939.909014:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202939.909015:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202939.909017:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187769344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202939.909022:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187769472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202939.909034:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202939.909040:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202939/real 1788202939] req@ffff8e828793f480 x1875062187769472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202939.909047:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187769472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202939.909053:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202939.909054:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202939.909056:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187769472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202939.909059:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187769600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202939.909069:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202939.909075:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202939/real 1788202939] req@ffff8e828793ca80 x1875062187769600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202939.909081:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187769600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202939.909086:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202939.909088:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202939.909090:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187769600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202940.612550:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202941.636493:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202942.660466:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202943.684458:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202944.708542:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202945.028414:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202945.028427:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202945.028432:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202945.028444:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4400 00000100:00080000:1.0:1788202945.028456:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202945.028464:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202945.028509:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187769728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202945.028532:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202945.028536:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202945.028537:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202945.028541:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4400 00000100:00080000:1.0:1788202945.028543:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202945.028546:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202945.028555:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202945.028566:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187769856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202945.028575:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202945.028577:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202945.028578:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202945.028582:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4400 00000100:00080000:1.0:1788202945.028586:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788202945.028587:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187769728:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202945.028590:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202945.028605:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202945.028615:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187769984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202945.028621:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202945.028624:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202945.028626:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202945.028628:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4400 00000100:00080000:1.0:1788202945.028631:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202945.028633:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202945.028637:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202945.028644:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187770112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202945.028650:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202945.028653:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202945.028654:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202945.028657:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4400 00000100:00080000:1.0:1788202945.028661:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202945.028663:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202945.028667:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202945.028674:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187770240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202945.028686:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202945.028780:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202945/real 1788202945] req@ffff8e828d720380 x1875062187769728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202945.028803:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187769728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202945.028817:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202945.028822:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202945.028827:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187769728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202945.028858:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187769856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202945.028886:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202945.028901:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202945/real 1788202945] req@ffff8e828d722300 x1875062187769856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202945.028915:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187769856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202945.028925:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202945.028928:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202945.028932:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187769856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202945.028939:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187769984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202945.028961:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202945.028973:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202945/real 1788202945] req@ffff8e828d722d80 x1875062187769984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202945.028986:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187769984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202945.029002:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202945.029005:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202945.029009:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187769984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202945.029016:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187770112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202945.029037:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202945.029049:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202945/real 1788202945] req@ffff8e828d721180 x1875062187770112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202945.029061:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187770112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202945.029069:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202945.029073:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202945.029076:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187770112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202945.029083:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187770240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202945.029103:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202945.029115:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202945/real 1788202945] req@ffff8e828d720a80 x1875062187770240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202970 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202945.029127:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187770240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202970 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202945.029135:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202945.029139:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202945.029143:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187770240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788202945.732432:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202946.756548:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202947.780454:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202948.804481:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202949.828518:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202950.148469:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202950.148477:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202950.148480:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202950.148488:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4405 00000100:00080000:0.0:1788202950.148498:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202950.148504:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202950.148536:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187770368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202950.148558:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202950.148561:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202950.148562:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202950.148565:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4405 00000100:00080000:0.0:1788202950.148567:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202950.148570:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202950.148577:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202950.148586:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187770496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202950.148598:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202950.148602:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202950.148604:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202950.148608:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4405 00000100:00080000:0.0:1788202950.148612:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202950.148616:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202950.148622:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202950.148636:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187770368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202950.148637:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187770624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202950.148668:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202950.148674:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202950.148688:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202950.148693:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4405 00000100:00080000:0.0:1788202950.148696:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202950.148699:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202950.148704:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202950.148716:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187770752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202950.148724:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202950.148727:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202950.148728:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202950.148730:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4405 00000100:00080000:0.0:1788202950.148732:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202950.148734:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202950.148738:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202950.148747:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187770880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202950.148757:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202950.148811:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202950/real 1788202950] req@ffff8e828793d500 x1875062187770368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202950.148835:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187770368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202950.148845:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202950.148849:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202950.148852:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187770368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202950.148859:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187770496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202950.148877:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202950.148887:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202950/real 1788202950] req@ffff8e828793d880 x1875062187770496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202950.148895:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187770496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202950.148901:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202950.148909:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202950.148911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187770496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202950.148936:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187770624:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202950.148950:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202950.148958:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202950/real 1788202950] req@ffff8e828793c380 x1875062187770624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202950.148969:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187770624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202950.148977:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202950.148981:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202950.148985:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187770624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202950.148993:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187770752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202950.149016:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202950.149026:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202950/real 1788202950] req@ffff8e828793fb80 x1875062187770752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202950.149035:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187770752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202950.149041:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202950.149043:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202950.149045:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187770752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202950.149049:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187770880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202950.149063:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202950.149070:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202950/real 1788202950] req@ffff8e828793ca80 x1875062187770880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202975 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202950.149082:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187770880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202975 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202950.149087:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202950.149089:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202950.149091:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187770880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202950.852502:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202951.876503:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202952.900500:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202953.924520:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202954.948488:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202955.268433:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202955.268444:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202955.268449:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202955.268466:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4410 00000100:00080000:0.0:1788202955.268480:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202955.268488:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202955.268532:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187771008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202955.268559:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202955.268562:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202955.268564:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202955.268567:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4410 00000100:00080000:0.0:1788202955.268571:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202955.268573:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202955.268583:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202955.268609:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187771136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202955.268622:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788202955.268627:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187771008:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202955.268628:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202955.268633:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202955.268638:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4410 00000100:00080000:0.0:1788202955.268643:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202955.268647:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202955.268653:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202955.268665:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187771264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202955.268676:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202955.268680:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202955.268682:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202955.268686:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4410 00000100:00080000:0.0:1788202955.268695:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202955.268699:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202955.268703:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788202955.268710:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202955.268715:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187771392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202955.268728:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202955.268730:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202955.268738:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202955.268742:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4410 00000100:00080000:0.0:1788202955.268744:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202955.268747:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202955.268750:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202955.268758:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187771520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202955.268774:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202955/real 1788202955] req@ffff8e828793f480 x1875062187771008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202955.268799:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187771008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202955.268812:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202955.268817:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202955.268821:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187771008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202955.268848:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187771136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202955.268868:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202955.268878:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202955/real 1788202955] req@ffff8e828793dc00 x1875062187771136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202955.268888:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187771136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202955.268898:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202955.268902:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202955.268906:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187771136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202955.268913:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187771264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202955.268935:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202955.268948:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202955/real 1788202955] req@ffff8e828793c000 x1875062187771264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202955.268960:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187771264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202955.268966:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202955.268968:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202955.268971:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187771264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202955.268975:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187771392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202955.268992:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202955.269010:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202955/real 1788202955] req@ffff8e828793e300 x1875062187771392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202955.269023:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187771392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202955.269031:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202955.269035:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202955.269039:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187771392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202955.269046:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187771520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202955.269068:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202955.269078:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202955/real 1788202955] req@ffff8e828793c700 x1875062187771520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202980 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202955.269086:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187771520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202980 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202955.269092:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202955.269094:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202955.269096:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187771520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202955.972489:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202956.996497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202958.020427:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202959.044487:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202960.068548:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202960.388445:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202960.388453:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202960.388458:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202960.388467:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4415 00000100:00080000:0.0:1788202960.388476:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202960.388485:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202960.388536:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187771648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202960.388558:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202960.388560:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202960.388562:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202960.388565:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4415 00000100:00080000:0.0:1788202960.388567:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202960.388571:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202960.388579:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202960.388589:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187771776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202960.388598:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202960.388603:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788202960.388605:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187771648:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202960.388606:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202960.388619:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4415 00000100:00080000:0.0:1788202960.388622:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202960.388625:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202960.388628:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202960.388637:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187771904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202960.388643:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202960.388646:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202960.388647:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202960.388649:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4415 00000100:00080000:0.0:1788202960.388652:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202960.388655:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202960.388659:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202960.388672:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187772032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202960.388683:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202960.388686:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202960.388689:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788202960.388700:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202960.388702:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4415 00000100:00080000:0.0:1788202960.388713:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202960.388721:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202960.388728:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202960.388756:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202960/real 1788202960] req@ffff8e828793ce00 x1875062187771648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202960.388758:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187772160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202960.388838:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187771648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202960.388851:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202960.388855:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202960.388858:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187771648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202960.388884:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187771776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202960.388907:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202960.388916:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202960/real 1788202960] req@ffff8e828793f800 x1875062187771776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202960.388925:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187771776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202960.388931:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202960.388934:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202960.388936:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187771776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202960.388940:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187771904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202960.388951:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202960.388957:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202960/real 1788202960] req@ffff8e828793e680 x1875062187771904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202960.388965:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187771904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202960.388970:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202960.388972:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202960.388974:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187771904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202960.388978:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187772032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202960.388990:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202960.389004:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202960/real 1788202960] req@ffff8e828793f100 x1875062187772032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202960.389022:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187772032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202960.389029:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202960.389033:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202960.389036:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187772032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202960.389042:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187772160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202960.389066:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202960.389074:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202960/real 1788202960] req@ffff8e828793c700 x1875062187772160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202960.389082:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187772160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202960.389087:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202960.389090:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202960.389092:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187772160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788202961.092504:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202962.116523:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202963.140464:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788202964.164465:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202965.188480:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202965.508442:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202965.508450:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202965.508454:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202965.508464:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4420 00000100:00080000:1.0:1788202965.508474:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202965.508483:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202965.508521:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187772288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202965.508574:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202965.508578:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202965.508580:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202965.508586:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4420 00000100:00080000:1.0:1788202965.508590:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202965.508595:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202965.508606:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202965.508617:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187772288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788202965.508624:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187772416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202965.508638:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202965.508642:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202965.508644:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202965.508649:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4420 00000100:00080000:1.0:1788202965.508653:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202965.508658:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202965.508664:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202965.508677:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187772544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202965.508688:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202965.508691:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202965.508695:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788202965.508696:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788202965.508700:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4420 00000100:00080000:1.0:1788202965.508706:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202965.508711:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202965.508717:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202965.508730:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187772672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202965.508742:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202965.508745:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788202965.508751:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202965/real 1788202965] req@ffff8e828d720a80 x1875062187772288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202965.508751:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202965.508760:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4420 00000100:00080000:1.0:1788202965.508767:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202965.508774:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202965.508780:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202965.508784:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187772288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788202965.508793:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187772800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202965.508795:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202965.508804:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202965.508810:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187772288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202965.508831:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187772416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202965.508849:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202965.508859:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202965/real 1788202965] req@ffff8e828d721180 x1875062187772416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202965.508868:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187772416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202965.508879:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202965.508881:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202965.508884:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187772416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202965.508888:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187772544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202965.508907:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202965.508920:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202965/real 1788202965] req@ffff8e828d722d80 x1875062187772544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202965.508932:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187772544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202965.508941:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202965.508945:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202965.508949:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187772544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202965.508956:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187772672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202965.508980:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202965.508996:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202965/real 1788202965] req@ffff8e828d722300 x1875062187772672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202965.509010:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187772672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202965.509020:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202965.509023:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202965.509026:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187772672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788202965.509031:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187772800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202965.509050:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202965.509064:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202965/real 1788202965] req@ffff8e828d720380 x1875062187772800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202965.509076:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187772800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202965.509081:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202965.509084:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202965.509087:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187772800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202966.212496:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788202967.236453:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202968.260495:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202969.284455:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202970.308490:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202970.628457:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202970.628465:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202970.628469:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202970.628478:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4425 00000100:00080000:0.0:1788202970.628487:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202970.628492:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202970.628525:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187772928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202970.628556:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202970.628559:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202970.628561:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202970.628564:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4425 00000100:00080000:0.0:1788202970.628567:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202970.628570:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202970.628577:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202970.628593:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187773056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202970.628605:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788202970.628606:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187772928:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202970.628608:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202970.628612:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202970.628617:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4425 00000100:00080000:0.0:1788202970.628619:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202970.628621:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202970.628625:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202970.628633:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187773184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202970.628640:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202970.628643:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202970.628644:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202970.628647:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4425 00000100:00080000:0.0:1788202970.628651:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202970.628655:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202970.628665:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202970.628678:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187773312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202970.628697:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202970.628701:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202970.628704:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202970.628709:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4425 00000100:00080000:0.0:1788202970.628712:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202970.628716:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202970.628721:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202970.628722:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187773056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202970.628735:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187773440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202970.628766:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187773184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202970.628786:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187773312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202970.628810:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187773440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202971.332494:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202972.356575:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202973.380560:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788202973.700556:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788202973.700593:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788202973.700609:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202973.700650:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202973.700660:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202973.700670:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788202973.700676:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:2.0:1788202973.700709:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202970/real 1788202973] req@ffff8e828793ca80 x1875062187773440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202973.700737:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187773440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202973.700750:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202973.700755:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202973.700760:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187773440:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202973.700770:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202970/real 1788202973] req@ffff8e828793c000 x1875062187773056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202973.700784:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187773056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202973.700793:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202973.700797:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202973.700801:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187773056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202973.700811:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202970/real 1788202973] req@ffff8e828793dc00 x1875062187773184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202973.700822:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187773184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788202995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202973.700830:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202973.700834:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202973.700838:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187773184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202973.700845:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202970/real 1788202973] req@ffff8e828793f480 x1875062187773312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202973.700872:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187773312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788202995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202973.700880:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202973.700884:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202973.700886:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187773312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202973.700893:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202970/real 1788202973] req@ffff8e828793e300 x1875062187772928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202973.700904:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187772928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788202995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202973.700911:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202973.700913:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202973.700916:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187772928:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788202974.404542:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202975.428503:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202975.748399:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202975.748408:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202975.748411:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202975.748420:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4430 00000100:00080000:0.0:1788202975.748429:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202975.748435:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202975.748466:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187773568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202975.748493:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202975.748496:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202975.748497:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202975.748500:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4430 00000100:00080000:0.0:1788202975.748502:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202975.748506:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202975.748513:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202975.748521:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187773696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202975.748529:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202975.748532:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202975.748534:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202975.748538:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4430 00000100:00080000:0.0:1788202975.748543:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788202975.748548:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187773568:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202975.748560:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202975.748567:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202975.748583:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187773824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202975.748600:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202975.748604:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202975.748606:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202975.748610:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4430 00000100:00080000:0.0:1788202975.748614:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788202975.748622:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788202975.748625:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202975.748633:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202975.748647:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187773952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202975.748658:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202975.748662:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202975.748664:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202975.748670:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4430 00000100:00100000:2.0:1788202975.748671:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202975/real 1788202975] req@ffff8e828793fb80 x1875062187773568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202975.748675:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202975.748683:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202975.748692:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202975.748697:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187773568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202975.748709:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187774080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202975.748711:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202975.748721:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202975.748729:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187773568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202975.748739:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187773696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202975.748767:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202975.748782:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202975/real 1788202975] req@ffff8e828793c380 x1875062187773696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202975.748796:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187773696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202975.748823:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202975.748827:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202975.748831:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187773696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202975.748858:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187773824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202975.748882:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202975.748895:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202975/real 1788202975] req@ffff8e828793d880 x1875062187773824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202975.748908:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187773824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202975.748916:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202975.748920:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202975.748923:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187773824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202975.748930:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187773952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202975.748953:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202975.748966:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202975/real 1788202975] req@ffff8e828793d500 x1875062187773952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202975.748980:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187773952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202975.748989:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202975.748992:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202975.748995:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187773952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202975.749002:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187774080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202975.749023:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202975.749036:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202975/real 1788202975] req@ffff8e828793e300 x1875062187774080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202975.749049:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187774080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202975.749058:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202975.749061:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202975.749064:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187774080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202976.452445:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202977.476450:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202978.500443:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202979.524458:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202980.548464:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788202980.868449:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202980.868459:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788202980.868465:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202980.868475:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4435 00000100:00080000:1.0:1788202980.868486:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202980.868492:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788202980.868536:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187774208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788202980.868567:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202980.868570:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788202980.868572:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202980.868576:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4435 00000100:00080000:1.0:1788202980.868579:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202980.868583:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202980.868593:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202980.868607:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187774336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202980.868614:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187774208:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202980.868624:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202980.868628:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788202980.868631:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202980.868636:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4435 00000100:00080000:1.0:1788202980.868640:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202980.868645:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202980.868651:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788202980.868662:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187774464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202980.868678:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788202980.868683:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202980.868688:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788202980.868690:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202980.868696:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4435 00000100:00080000:1.0:1788202980.868700:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788202980.868705:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202980.868710:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788202980.868725:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187774592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202980.868726:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202980/real 1788202980] req@ffff8e828d720e00 x1875062187774208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788202980.868739:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788202980.868748:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788202980.868752:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788202980.868758:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4435 00000100:00100000:2.0:1788202980.868761:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187774208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202980.868771:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788202980.868772:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202980.868775:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202980.868781:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187774208:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788202980.868784:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788202980.868792:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202980.868805:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187774336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788202980.868807:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187774720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788202980.868832:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202980.868849:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202980/real 1788202980] req@ffff8e828d723800 x1875062187774336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202980.868863:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187774336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202980.868873:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202980.868877:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202980.868880:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187774336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202980.868887:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187774464:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202980.868905:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202980.868913:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202980/real 1788202980] req@ffff8e828d721500 x1875062187774464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202980.868921:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187774464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202980.868931:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202980.868933:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202980.868936:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187774464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202980.868939:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187774592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202980.868950:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202980.868956:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202980/real 1788202980] req@ffff8e828d720000 x1875062187774592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202980.868963:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187774592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202980.868969:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202980.868970:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202980.868972:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187774592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202980.868983:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187774720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202980.868994:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202980.869000:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202980/real 1788202980] req@ffff8e828d722680 x1875062187774720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202980.869007:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187774720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202980.869012:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202980.869013:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202980.869016:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187774720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202981.572445:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202982.596456:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202983.620493:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202984.644431:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202985.668479:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202985.988431:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202985.988438:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202985.988442:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202985.988451:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4441 00000100:00080000:0.0:1788202985.988459:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202985.988465:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202985.988495:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187774848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202985.988520:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202985.988524:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202985.988526:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202985.988530:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4441 00000100:00080000:0.0:1788202985.988534:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202985.988539:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202985.988549:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202985.988564:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187774976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202985.988566:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187774848:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202985.988580:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202985.988582:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202985.988584:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202985.988587:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4441 00000100:00080000:0.0:1788202985.988590:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202985.988592:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202985.988596:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202985.988604:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187775104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202985.988610:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202985.988613:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788202985.988614:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202985.988617:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4441 00000100:00080000:0.0:1788202985.988619:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202985.988622:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202985.988624:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788202985.988630:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202985.988644:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187775232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202985.988650:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202985.988653:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202985.988655:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202985.988658:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4441 00000100:00080000:0.0:1788202985.988662:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202985.988664:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202985.988668:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202985.988681:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187775360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202985.988687:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202985/real 1788202985] req@ffff8e828793f480 x1875062187774848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202985.988706:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187774848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202985.988714:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202985.988717:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202985.988720:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187774848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202985.988740:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187774976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202985.988758:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202985.988767:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202985/real 1788202985] req@ffff8e828793dc00 x1875062187774976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202985.988775:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187774976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202985.988780:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202985.988782:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202985.988785:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187774976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202985.988788:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187775104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202985.988809:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202985.988816:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202985/real 1788202985] req@ffff8e828793c000 x1875062187775104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202985.988824:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187775104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202985.988829:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202985.988831:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202985.988833:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187775104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202985.988837:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187775232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202985.988847:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202985.988853:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202985/real 1788202985] req@ffff8e828793ca80 x1875062187775232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202985.988860:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187775232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202985.988865:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202985.988867:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202985.988869:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187775232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202985.988872:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187775360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202985.988881:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202985.988887:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202985/real 1788202985] req@ffff8e828793c700 x1875062187775360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202985.988894:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187775360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202985.988899:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202985.988901:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202985.988903:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187775360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202986.692447:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202987.716499:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202988.740497:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202989.764503:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202990.788494:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788202991.108429:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202991.108441:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788202991.108445:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202991.108455:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4446 00000100:00080000:0.0:1788202991.108466:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202991.108473:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788202991.108514:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187775488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202991.108543:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202991.108547:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788202991.108549:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202991.108553:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4446 00000100:00080000:0.0:1788202991.108557:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202991.108561:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202991.108571:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202991.108585:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187775616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202991.108595:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202991.108598:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788202991.108600:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202991.108604:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4446 00000100:00080000:0.0:1788202991.108607:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202991.108611:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202991.108616:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202991.108628:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187775744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202991.108638:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202991.108644:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788202991.108646:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187775488:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788202991.108647:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202991.108656:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4446 00000100:00080000:0.0:1788202991.108660:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202991.108664:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202991.108670:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788202991.108682:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187775872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788202991.108692:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788202991.108696:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788202991.108698:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788202991.108702:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4446 00000100:00080000:0.0:1788202991.108708:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788202991.108712:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788202991.108717:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788202991.108719:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202991.108735:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187776000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788202991.108766:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202991/real 1788202991] req@ffff8e828793f100 x1875062187775488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202991.108787:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187775488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202991.108799:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202991.108804:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202991.108809:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187775488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202991.108820:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187775616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202991.108846:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202991.108860:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202991/real 1788202991] req@ffff8e828793e680 x1875062187775616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202991.108874:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187775616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202991.108883:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202991.108887:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202991.108890:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187775616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202991.108915:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187775744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788202991.108938:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202991.108951:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202991/real 1788202991] req@ffff8e828793f800 x1875062187775744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202991.108997:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187775744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202991.109007:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202991.109011:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202991.109015:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187775744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788202991.109022:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187775872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202991.109046:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202991.109059:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202991/real 1788202991] req@ffff8e828793ce00 x1875062187775872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202991.109072:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187775872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202991.109081:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202991.109084:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202991.109088:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187775872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788202991.109105:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187776000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788202991.109128:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788202991.109140:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202991/real 1788202991] req@ffff8e828793c700 x1875062187776000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203016 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788202991.109153:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187776000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203016 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788202991.109162:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788202991.109166:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788202991.109169:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187776000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788202991.812523:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202992.836525:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202993.860473:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202994.884489:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202995.908492:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788202996.228455:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202996.228463:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788202996.228466:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202996.228474:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4451 00000100:00080000:2.0:1788202996.228484:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202996.228489:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788202996.228523:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187776128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202996.228550:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202996.228553:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788202996.228554:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202996.228557:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4451 00000100:00080000:2.0:1788202996.228559:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202996.228561:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202996.228568:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788202996.228577:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187776256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202996.228589:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202996.228591:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788202996.228593:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202996.228596:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4451 00000100:00080000:2.0:1788202996.228599:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202996.228601:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202996.228605:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788202996.228615:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187776128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788202996.228616:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187776384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202996.228638:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202996.228641:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788202996.228642:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202996.228646:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4451 00000100:00080000:2.0:1788202996.228649:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202996.228651:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202996.228657:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202996.228672:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187776512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788202996.228683:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788202996.228687:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788202996.228689:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788202996.228695:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4451 00000100:00080000:2.0:1788202996.228701:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788202996.228706:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788202996.228712:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788202996.228726:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187776640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788202996.228727:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202996.228792:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202996/real 1788202996] req@ffff8e82894e5f80 x1875062187776128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202996.228817:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187776128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202996.228832:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202996.228836:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202996.228840:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187776128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788202996.228847:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187776256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202996.228874:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202996.228918:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202996/real 1788202996] req@ffff8e82894e4380 x1875062187776256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202996.228933:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187776256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202996.228945:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202996.228948:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202996.228950:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187776256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202996.228975:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187776384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788202996.228992:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202996.229000:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202996/real 1788202996] req@ffff8e82894e5c00 x1875062187776384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202996.229008:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187776384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202996.229014:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202996.229016:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202996.229019:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187776384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788202996.229022:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187776512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202996.229033:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202996.229041:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202996/real 1788202996] req@ffff8e82894e6d80 x1875062187776512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202996.229052:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187776512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202996.229061:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202996.229064:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202996.229068:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187776512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788202996.229075:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187776640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788202996.229095:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788202996.229105:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788202996/real 1788202996] req@ffff8e82894e7100 x1875062187776640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203021 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788202996.229113:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187776640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203021 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788202996.229119:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788202996.229121:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788202996.229123:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187776640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788202996.932571:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788202997.956469:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788202998.980498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203000.004491:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203001.028422:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203001.348394:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203001.348406:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203001.348410:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203001.348421:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4456 00000100:00080000:0.0:1788203001.348433:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203001.348439:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203001.348477:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187776768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203001.348498:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203001.348501:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203001.348503:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203001.348506:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4456 00000100:00080000:0.0:1788203001.348509:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203001.348512:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203001.348519:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203001.348528:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187776896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203001.348534:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203001.348536:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203001.348537:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203001.348539:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4456 00000100:00080000:0.0:1788203001.348541:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203001.348544:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203001.348547:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203001.348553:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187777024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203001.348559:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203001.348561:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203001.348562:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203001.348564:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4456 00000100:00080000:0.0:1788203001.348566:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203001.348568:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203001.348571:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203001.348575:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187776768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203001.348579:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187777152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203001.348591:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203001.348593:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203001.348594:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203001.348596:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4456 00000100:00080000:0.0:1788203001.348598:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203001.348600:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203001.348603:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203001.348610:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187777280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203001.348647:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203001.348685:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203001/real 1788203001] req@ffff8e828793ca80 x1875062187776768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203001.348700:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187776768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203001.348709:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203001.348712:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203001.348716:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187776768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203001.348722:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187776896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203001.348749:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203001.348758:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203001/real 1788203001] req@ffff8e828793c000 x1875062187776896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203001.348768:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187776896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203001.348774:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203001.348784:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203001.348801:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187776896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203001.348808:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187777024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203001.348826:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203001.348834:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203001/real 1788203001] req@ffff8e828793dc00 x1875062187777024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203001.348843:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187777024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203001.348848:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203001.348853:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203001.348856:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187777024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203001.348883:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187777152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203001.348896:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203001.348903:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203001/real 1788203001] req@ffff8e828793f480 x1875062187777152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203001.348911:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187777152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203001.348916:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203001.348919:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203001.348923:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187777152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203001.348927:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187777280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203001.348938:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203001.348945:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203001/real 1788203001] req@ffff8e828793e300 x1875062187777280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203001.348952:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187777280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203001.348958:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203001.348960:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203001.348963:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187777280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203002.052440:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203003.076523:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203004.100472:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203005.124479:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203006.148471:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203006.468423:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203006.468433:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203006.468438:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203006.468449:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4461 00000100:00080000:2.0:1788203006.468459:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203006.468466:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203006.468502:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187777408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203006.468530:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203006.468534:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203006.468536:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203006.468541:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4461 00000100:00080000:2.0:1788203006.468545:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203006.468550:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203006.468560:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203006.468566:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187777408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203006.468578:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187777536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203006.468590:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203006.468594:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203006.468596:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203006.468601:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4461 00000100:00080000:2.0:1788203006.468605:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203006.468609:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203006.468614:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203006.468627:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187777664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203006.468629:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203006.468640:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203006.468645:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203006.468647:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203006.468651:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4461 00000100:00080000:2.0:1788203006.468655:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203006.468660:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203006.468667:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203006.468668:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203006/real 1788203006] req@ffff8e82894e6300 x1875062187777408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203006.468683:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187777792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203006.468692:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187777408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203006.468711:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203006.468720:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203006.468723:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203006.468731:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4461 00000100:00080000:0.0:1788203006.468732:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203006.468737:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203006.468742:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:2.0:1788203006.468747:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203006.468753:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187777408:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788203006.468756:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203006.468773:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187777920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203006.468787:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187777536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203006.468815:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203006.468830:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203006/real 1788203006] req@ffff8e82894e7800 x1875062187777536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203006.468844:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187777536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203006.468853:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203006.468856:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203006.468860:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187777536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203006.468866:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187777664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203006.468897:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203006.468910:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203006/real 1788203006] req@ffff8e82894e4a80 x1875062187777664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203006.468924:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187777664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203006.468933:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203006.468936:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203006.468940:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187777664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203006.468947:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187777792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203006.468974:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203006.468985:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203006/real 1788203006] req@ffff8e82894e4e00 x1875062187777792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203006.468993:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187777792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203006.468998:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203006.469001:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203006.469003:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187777792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203006.469022:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187777920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203006.469043:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203006.469055:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203006/real 1788203006] req@ffff8e82894e4000 x1875062187777920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203006.469067:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187777920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203006.469075:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203006.469078:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203006.469082:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187777920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203007.172474:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203008.196435:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203009.220451:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203010.244462:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203011.268448:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203011.588425:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203011.588434:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203011.588437:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203011.588445:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4466 00000100:00080000:2.0:1788203011.588454:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203011.588460:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203011.588492:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187778048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203011.588515:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203011.588518:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203011.588519:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203011.588521:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4466 00000100:00080000:2.0:1788203011.588524:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203011.588526:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203011.588533:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203011.588541:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187778176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203011.588547:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203011.588551:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203011.588553:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203011.588555:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4466 00000100:00080000:2.0:1788203011.588558:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203011.588572:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203011.588574:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203011.588581:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187778304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203011.588587:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203011.588589:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203011.588592:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788203011.588594:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187778048:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203011.588596:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4466 00000100:00080000:2.0:1788203011.588600:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203011.588603:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203011.588606:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203011.588631:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187778432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203011.588644:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203011.588647:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203011.588648:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203011.588650:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4466 00000100:00080000:2.0:1788203011.588654:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203011.588657:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203011.588661:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203011.588668:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187778560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203011.588685:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203011.588722:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203011/real 1788203011] req@ffff8e82894e6a00 x1875062187778048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203011.588744:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187778048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203011.588752:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203011.588755:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203011.588758:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187778048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203011.588764:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187778176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203011.588779:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203011.588788:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203011/real 1788203011] req@ffff8e82894e7480 x1875062187778176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203011.588796:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187778176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203011.588802:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203011.588804:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203011.588806:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187778176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203011.588824:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187778304:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203011.588835:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203011.588841:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203011/real 1788203011] req@ffff8e82894e6680 x1875062187778304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203011.588848:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187778304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203011.588853:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203011.588855:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203011.588857:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187778304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203011.588861:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187778432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203011.588872:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203011.588878:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203011/real 1788203011] req@ffff8e82894e5880 x1875062187778432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203011.588885:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187778432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203011.588890:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203011.588892:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203011.588900:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187778432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203011.588905:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187778560:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203011.588916:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203011.588923:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203011/real 1788203011] req@ffff8e82894e4000 x1875062187778560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203011.588933:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187778560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203011.588941:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203011.588944:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203011.588947:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187778560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203012.292509:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203013.316501:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203014.340467:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203015.364474:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203016.388542:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203016.708400:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203016.708409:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203016.708413:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203016.708422:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4471 00000100:00080000:2.0:1788203016.708431:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203016.708438:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203016.708473:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187778688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203016.708500:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203016.708502:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203016.708504:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203016.708507:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4471 00000100:00080000:2.0:1788203016.708509:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203016.708511:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203016.708534:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203016.708549:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187778816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203016.708561:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203016.708565:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203016.708567:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788203016.708573:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187778688:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203016.708573:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4471 00000100:00080000:2.0:1788203016.708582:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203016.708586:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203016.708591:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203016.708603:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187778944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203016.708611:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203016.708613:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203016.708615:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203016.708617:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4471 00000100:00080000:2.0:1788203016.708620:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203016.708622:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203016.708626:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788203016.708634:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203016.708635:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187779072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203016.708644:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203016.708647:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203016.708648:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203016.708651:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4471 00000100:00080000:2.0:1788203016.708654:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203016.708656:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203016.708659:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203016.708664:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187779200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203016.708689:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203016/real 1788203016] req@ffff8e82894e4e00 x1875062187778688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203016.708703:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187778688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203016.708710:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203016.708713:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203016.708716:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187778688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203016.708721:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187778816:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203016.708736:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203016.708745:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203016/real 1788203016] req@ffff8e82894e4a80 x1875062187778816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203016.708753:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187778816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203016.708758:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203016.708760:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203016.708762:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187778816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203016.708779:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187778944:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203016.708789:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203016.708795:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203016/real 1788203016] req@ffff8e82894e7800 x1875062187778944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203016.708803:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187778944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203016.708812:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203016.708814:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203016.708816:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187778944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203016.708820:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187779072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203016.708829:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203016.708835:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203016/real 1788203016] req@ffff8e82894e6300 x1875062187779072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203016.708842:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187779072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203016.708847:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203016.708849:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203016.708850:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187779072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203016.708854:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187779200:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203016.708863:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203016.708869:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203016/real 1788203016] req@ffff8e82894e7100 x1875062187779200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203016.708876:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187779200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203016.708880:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203016.708882:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203016.708884:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187779200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203017.412549:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203018.436557:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203019.460516:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203020.484542:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203021.508543:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203021.828501:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203021.828509:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203021.828514:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203021.828524:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4476 00000100:00080000:0.0:1788203021.828536:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203021.828544:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203021.828582:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187779328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203021.828647:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187779328:192.168.201.121@tcp:250:kworker.0 00000800:00000100:2.0:1788203021.828753:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203021.828809:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203021.828814:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203021.828816:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203021.828821:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4476 00000100:00080000:0.0:1788203021.828827:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203021.828833:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203021.828847:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203021.828895:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187779456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203021.828961:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203021/real 1788203021] req@ffff8e828793d500 x1875062187779328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203021.828998:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187779328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203021.829007:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203021.829011:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203021.829017:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187779328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203021.829048:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187779456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203021.829077:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203021.829093:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203021/real 1788203021] req@ffff8e828793d880 x1875062187779456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203021.829107:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187779456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203021.829117:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203021.829120:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203021.829124:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187779456:192.168.201.121@tcp:38:kworker.0 00000100:00080000:0.0:1788203021.829177:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203021.829181:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203021.829183:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203021.829187:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4476 00000100:00080000:0.0:1788203021.829190:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203021.829195:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203021.829200:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203021.829210:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187779584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203021.829218:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203021.829221:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203021.829222:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203021.829225:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4476 00000100:00080000:0.0:1788203021.829227:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203021.829229:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203021.829232:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203021.829240:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187779712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203021.829245:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203021.829247:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203021.829248:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203021.829251:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4476 00000100:00080000:0.0:1788203021.829253:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203021.829255:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203021.829258:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203021.829263:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187779840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203021.829341:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187779584:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203021.829362:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203021.829379:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203021/real 1788203021] req@ffff8e828793c380 x1875062187779584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203021.829390:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187779584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203021.829397:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203021.829399:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203021.829402:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187779584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203021.829406:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187779712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203021.829424:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203021.829449:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203021/real 1788203021] req@ffff8e828793fb80 x1875062187779712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203021.829463:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187779712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203021.829489:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203021.829493:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203021.829497:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187779712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203021.829505:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187779840:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203021.829522:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203021.829531:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203021/real 1788203021] req@ffff8e828793d880 x1875062187779840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203021.829539:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187779840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203021.829545:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203021.829548:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203021.829550:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187779840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203022.532541:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203023.556513:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203024.580555:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203025.604527:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203026.628495:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203026.948425:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203026.948436:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203026.948440:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203026.948450:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4482 00000100:00080000:0.0:1788203026.948462:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203026.948470:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203026.948506:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187779968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203026.948534:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203026.948537:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203026.948539:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203026.948544:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4482 00000100:00080000:0.0:1788203026.948547:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203026.948552:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203026.948561:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203026.948574:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187780096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203026.948582:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187779968:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203026.948589:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203026.948594:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203026.948596:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203026.948601:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4482 00000100:00080000:0.0:1788203026.948605:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203026.948609:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203026.948615:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203026.948627:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187780224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203026.948637:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203026.948641:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1788203026.948644:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203026.948646:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203026.948653:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4482 00000100:00080000:0.0:1788203026.948658:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203026.948663:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203026.948669:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203026.948681:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187780352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203026.948691:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203026/real 1788203026] req@ffff8e828793d500 x1875062187779968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203026.948692:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203026.948702:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203026.948706:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203026.948712:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4482 00000100:00100000:2.0:1788203026.948716:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187779968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203026.948717:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203026.948726:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:2.0:1788203026.948728:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203026.948734:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000008:00100000:0.0:1788203026.948736:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203026.948738:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187779968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203026.948754:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187780480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203026.948760:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187780096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203026.948784:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203026.948796:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203026/real 1788203026] req@ffff8e828793e300 x1875062187780096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203026.948805:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187780096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203026.948811:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203026.948813:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203026.948815:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187780096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203026.948819:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187780224:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203026.948831:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203026.948837:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203026/real 1788203026] req@ffff8e828793f480 x1875062187780224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203026.948844:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187780224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203026.948850:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203026.948852:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203026.948853:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187780224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203026.948857:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187780352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203026.948871:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203026.948882:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203026/real 1788203026] req@ffff8e828793dc00 x1875062187780352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203026.948895:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187780352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203026.948903:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203026.948907:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203026.948911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187780352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203026.948928:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187780480:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203026.948945:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203026.948956:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203026/real 1788203026] req@ffff8e828793c000 x1875062187780480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203026.948970:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187780480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203026.948978:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203026.948981:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203026.948985:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187780480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203027.652456:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203028.676505:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203029.700507:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203030.724499:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203031.748475:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203032.068446:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203032.068457:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203032.068462:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203032.068473:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4487 00000100:00080000:1.0:1788203032.068485:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203032.068493:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203032.068543:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187780608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203032.068565:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203032.068568:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203032.068569:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203032.068573:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4487 00000100:00080000:1.0:1788203032.068577:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203032.068582:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203032.068591:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203032.068611:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187780736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203032.068624:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788203032.068628:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187780608:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788203032.068629:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203032.068632:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203032.068640:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4487 00000100:00080000:1.0:1788203032.068643:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203032.068646:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203032.068650:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203032.068664:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187780864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203032.068675:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203032.068679:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203032.068682:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203032.068686:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4487 00000100:00080000:1.0:1788203032.068700:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203032.068707:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:2.0:1788203032.068708:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:1.0:1788203032.068713:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203032.068724:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187780992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203032.068732:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203032.068735:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203032.068736:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203032.068739:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4487 00000100:00080000:1.0:1788203032.068742:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203032.068745:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203032.068748:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203032.068757:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203032/real 1788203032] req@ffff8e828d722680 x1875062187780608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203032.068758:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187781120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203032.068800:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187780608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203032.068809:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203032.068813:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203032.068816:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187780608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203032.068836:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187780736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203032.068852:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203032.068868:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203032/real 1788203032] req@ffff8e828d720000 x1875062187780736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203032.068876:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187780736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203032.068886:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203032.068888:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203032.068891:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187780736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203032.068895:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187780864:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203032.068906:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203032.068913:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203032/real 1788203032] req@ffff8e828d721500 x1875062187780864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203032.068920:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187780864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203032.068925:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203032.068927:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203032.068929:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187780864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203032.068932:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187780992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203032.068943:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203032.068949:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203032/real 1788203032] req@ffff8e828d723800 x1875062187780992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203032.068955:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187780992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203032.068960:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203032.068961:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203032.068964:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187780992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203032.068968:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187781120:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203032.068978:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203032.068983:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203032/real 1788203032] req@ffff8e828d720e00 x1875062187781120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203057 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203032.068990:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187781120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203057 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203032.068995:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203032.068997:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203032.068999:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187781120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203032.772499:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203033.796486:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203034.820463:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203035.844478:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203036.868628:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203037.188405:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203037.188413:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203037.188417:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203037.188425:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4492 00000100:00080000:2.0:1788203037.188434:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203037.188440:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203037.188470:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187781248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203037.188493:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203037.188497:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203037.188499:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203037.188504:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4492 00000100:00080000:2.0:1788203037.188507:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203037.188513:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203037.188522:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203037.188544:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187781248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203037.188547:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187781376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203037.188557:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203037.188560:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203037.188562:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203037.188564:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4492 00000100:00080000:2.0:1788203037.188568:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203037.188571:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203037.188575:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203037.188583:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187781504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203037.188592:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203037.188595:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203037.188596:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203037.188599:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4492 00000100:00080000:2.0:1788203037.188602:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203037.188606:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203037.188611:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203037.188620:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187781632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203037.188630:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203037.188633:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203037.188635:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203037.188639:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4492 00000100:00080000:2.0:1788203037.188642:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788203037.188643:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187781376:192.168.201.121@tcp:38:kworker.0 00000100:00080000:2.0:1788203037.188646:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203037.188650:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203037.188662:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187781760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203037.188679:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187781504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203037.188702:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187781632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203037.188729:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187781760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203037.892491:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203038.916494:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203039.940554:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203040.260607:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788203040.260657:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788203040.260677:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203040.260733:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203040.260745:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203040.260755:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203040.260766:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:0.0:1788203040.260769:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203037/real 1788203040] req@ffff8e82894e5c00 x1875062187781376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203040.260799:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187781376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203040.260813:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203040.260819:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203040.260824:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187781376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203040.260834:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203037/real 1788203040] req@ffff8e82894e4380 x1875062187781504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203040.260847:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187781504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203040.260855:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203040.260858:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203040.260862:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187781504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203040.260869:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203037/real 1788203040] req@ffff8e82894e5f80 x1875062187781632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203040.260881:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187781632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203040.260888:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203040.260892:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203040.260896:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187781632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203040.260902:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203037/real 1788203040] req@ffff8e82894e6d80 x1875062187781248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203040.260915:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187781248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203040.260922:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203040.260926:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203040.260929:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187781248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203040.260960:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203037/real 1788203040] req@ffff8e82894e7100 x1875062187781760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203062 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203040.260973:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187781760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203062 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203040.260980:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203040.260984:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203040.260987:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187781760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203040.964448:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203041.988476:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203042.308395:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203042.308406:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203042.308409:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203042.308419:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4497 00000100:00080000:2.0:1788203042.308430:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203042.308437:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203042.308468:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187781888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203042.308492:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203042.308495:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203042.308497:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203042.308501:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4497 00000100:00080000:2.0:1788203042.308504:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203042.308508:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203042.308517:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203042.308529:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187782016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203042.308539:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788203042.308541:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187781888:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203042.308544:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203042.308548:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203042.308553:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4497 00000100:00080000:2.0:1788203042.308556:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203042.308560:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203042.308565:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203042.308577:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187782144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203042.308587:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203042.308590:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203042.308592:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203042.308596:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4497 00000100:00080000:2.0:1788203042.308599:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203042.308604:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203042.308609:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788203042.308611:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203042.308622:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187782272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203042.308631:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203042.308635:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203042.308636:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203042.308640:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4497 00000100:00080000:2.0:1788203042.308644:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203042.308648:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203042.308652:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203042/real 1788203042] req@ffff8e82894e6300 x1875062187781888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788203042.308655:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203042.308670:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187782400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203042.308681:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187781888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203042.308692:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203042.308697:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203042.308702:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187781888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203042.308724:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187782016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203042.308743:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203042.308754:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203042/real 1788203042] req@ffff8e82894e7800 x1875062187782016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203042.308765:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187782016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203042.308772:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203042.308775:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203042.308778:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187782016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203042.308783:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187782144:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203042.308800:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203042.308810:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203042/real 1788203042] req@ffff8e82894e4a80 x1875062187782144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203042.308820:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187782144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203042.308827:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203042.308830:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203042.308833:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187782144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203042.308838:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187782272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203042.308853:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203042.308861:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203042/real 1788203042] req@ffff8e82894e4e00 x1875062187782272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203042.308871:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187782272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203042.308878:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203042.308881:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203042.308884:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187782272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203042.308888:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187782400:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203042.308902:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203042.308910:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203042/real 1788203042] req@ffff8e82894e4000 x1875062187782400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203042.308919:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187782400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203042.308925:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203042.308928:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203042.308930:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187782400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203043.012497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203044.036468:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203045.060427:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203046.084438:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203047.108468:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203047.428427:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203047.428438:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203047.428442:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203047.428453:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4502 00000100:00080000:0.0:1788203047.428462:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203047.428470:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203047.428504:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187782528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203047.428529:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203047.428533:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203047.428535:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203047.428539:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4502 00000100:00080000:0.0:1788203047.428542:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203047.428546:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203047.428555:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203047.428570:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187782656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203047.428570:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187782528:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203047.428585:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203047.428588:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203047.428591:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203047.428594:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4502 00000100:00080000:0.0:1788203047.428597:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203047.428600:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203047.428605:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203047.428616:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187782784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203047.428625:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203047.428628:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203047.428630:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203047.428633:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4502 00000100:00080000:0.0:1788203047.428638:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788203047.428638:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203047.428646:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203047.428651:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203047.428662:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187782912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203047.428671:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203047.428674:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203047.428676:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788203047.428681:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203047/real 1788203047] req@ffff8e828793ca80 x1875062187782528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203047.428682:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4502 00000100:00080000:0.0:1788203047.428688:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203047.428695:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203047.428702:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203047.428708:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187782528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203047.428714:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187783040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203047.428722:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203047.428729:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203047.428733:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187782528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203047.428754:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187782656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203047.428773:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203047.428783:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203047/real 1788203047] req@ffff8e828793c700 x1875062187782656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203047.428792:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187782656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203047.428799:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203047.428802:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203047.428806:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187782656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203047.428811:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187782784:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203047.428825:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203047.428833:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203047/real 1788203047] req@ffff8e828793ce00 x1875062187782784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203047.428842:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187782784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203047.428849:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203047.428851:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203047.428854:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187782784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203047.428859:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187782912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203047.428874:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203047.428883:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203047/real 1788203047] req@ffff8e828793f800 x1875062187782912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203047.428899:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187782912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203047.428906:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203047.428908:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203047.428911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187782912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203047.428916:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187783040:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203047.428930:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203047.428939:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203047/real 1788203047] req@ffff8e828793e680 x1875062187783040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203047.428949:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187783040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203047.428956:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203047.428959:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203047.428962:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187783040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203048.132432:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203049.156460:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203050.180455:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203051.204433:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203052.228493:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203052.548448:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203052.548458:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203052.548463:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203052.548475:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4507 00000100:00080000:2.0:1788203052.548488:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203052.548496:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203052.548539:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187783168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203052.548566:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203052.548570:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203052.548572:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203052.548577:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4507 00000100:00080000:2.0:1788203052.548581:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203052.548587:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203052.548596:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203052.548608:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187783168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203052.548613:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187783296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203052.548627:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203052.548631:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203052.548633:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203052.548637:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4507 00000100:00080000:2.0:1788203052.548641:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203052.548645:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203052.548651:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203052.548664:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187783424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203052.548675:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788203052.548679:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203052.548683:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203052.548686:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203052.548692:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4507 00000100:00080000:2.0:1788203052.548697:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203052.548702:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203052.548708:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203052.548722:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187783552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203052.548729:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203052/real 1788203052] req@ffff8e82894e5880 x1875062187783168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203052.548738:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203052.548747:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203052.548750:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788203052.548757:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187783168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203052.548757:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4507 00000100:00080000:2.0:1788203052.548764:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203052.548773:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:0.0:1788203052.548775:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1788203052.548780:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788203052.548783:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203052.548791:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187783168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203052.548799:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187783680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203052.548817:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187783296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203052.548843:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203052.548865:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203052/real 1788203052] req@ffff8e82894e6680 x1875062187783296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203052.548884:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187783296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203052.548892:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203052.548896:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203052.548926:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187783296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203052.548933:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187783424:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203052.548955:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203052.548967:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203052/real 1788203052] req@ffff8e82894e7480 x1875062187783424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203052.548979:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187783424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203052.548988:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203052.548991:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203052.548994:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187783424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203052.549000:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187783552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203052.549018:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203052.549029:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203052/real 1788203052] req@ffff8e82894e6a00 x1875062187783552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203052.549039:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187783552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203052.549048:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203052.549051:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203052.549054:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187783552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203052.549060:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187783680:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203052.549077:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203052.549088:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203052/real 1788203052] req@ffff8e82894e4000 x1875062187783680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203052.549106:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187783680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203052.549114:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203052.549117:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203052.549120:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187783680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203053.252503:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203054.276456:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203055.300469:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203056.324469:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203057.348476:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203057.668398:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203057.668407:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203057.668410:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203057.668419:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4512 00000100:00080000:0.0:1788203057.668428:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203057.668441:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203057.668470:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187783808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203057.668492:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203057.668494:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203057.668496:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203057.668499:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4512 00000100:00080000:0.0:1788203057.668501:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203057.668504:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203057.668511:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203057.668529:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187783936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203057.668533:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187783808:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203057.668549:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203057.668551:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203057.668553:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203057.668557:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4512 00000100:00080000:0.0:1788203057.668559:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203057.668562:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203057.668565:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203057.668578:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187784064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203057.668584:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203057.668587:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203057.668588:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203057.668590:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4512 00000100:00080000:0.0:1788203057.668593:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788203057.668594:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203057.668598:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203057.668603:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203057.668611:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187784192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203057.668617:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203057.668620:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203057.668621:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203057.668623:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4512 00000100:00080000:0.0:1788203057.668627:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203057.668630:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203057.668634:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203057.668638:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203057/real 1788203057] req@ffff8e828793f100 x1875062187783808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203057.668644:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187784320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203057.668677:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187783808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203057.668691:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203057.668696:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203057.668701:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187783808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203057.668721:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187783936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203057.668740:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203057.668753:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203057/real 1788203057] req@ffff8e828793e680 x1875062187783936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203057.668767:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187783936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203057.668776:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203057.668780:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203057.668784:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187783936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203057.668790:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187784064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203057.668827:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203057.668840:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203057/real 1788203057] req@ffff8e828793f800 x1875062187784064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203057.668853:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187784064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203057.668859:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203057.668861:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203057.668863:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187784064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203057.668868:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187784192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203057.668884:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203057.668891:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203057/real 1788203057] req@ffff8e828793ce00 x1875062187784192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203057.668898:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187784192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203057.668903:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203057.668905:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203057.668908:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187784192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203057.668913:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187784320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203057.668931:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203057.668943:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203057/real 1788203057] req@ffff8e828793c700 x1875062187784320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203057.668956:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187784320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203057.668964:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203057.668967:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203057.668971:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187784320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203058.372484:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203059.396540:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203060.420487:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203061.444491:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203062.468476:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203062.788436:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203062.788444:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203062.788448:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203062.788457:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4517 00000100:00080000:0.0:1788203062.788467:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203062.788474:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203062.788517:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187784448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203062.788540:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203062.788543:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203062.788545:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203062.788550:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4517 00000100:00080000:0.0:1788203062.788554:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203062.788559:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203062.788568:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203062.788581:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187784576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203062.788589:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187784448:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203062.788592:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203062.788606:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203062.788608:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203062.788611:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4517 00000100:00080000:0.0:1788203062.788615:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203062.788618:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203062.788623:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203062.788633:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187784704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203062.788641:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203062.788644:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203062.788646:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203062.788649:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4517 00000800:00000100:2.0:1788203062.788649:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203062.788653:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203062.788656:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203062.788661:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203062.788669:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187784832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203062.788677:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203062.788681:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203062.788683:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203062.788686:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4517 00000100:00080000:0.0:1788203062.788688:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203062.788689:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203062/real 1788203062] req@ffff8e828793ca80 x1875062187784448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203062.788692:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203062.788697:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203062.788706:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187784960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203062.788717:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187784448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203062.788740:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203062.788745:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203062.788748:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187784448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203062.788771:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187784576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203062.788793:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203062.788803:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203062/real 1788203062] req@ffff8e828793c000 x1875062187784576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203062.788811:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187784576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203062.788821:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203062.788824:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203062.788826:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187784576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203062.788830:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187784704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203062.788841:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203062.788847:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203062/real 1788203062] req@ffff8e828793dc00 x1875062187784704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203062.788854:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187784704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203062.788859:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203062.788861:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203062.788863:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187784704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203062.788866:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187784832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203062.788876:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203062.788882:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203062/real 1788203062] req@ffff8e828793f480 x1875062187784832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203062.788889:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187784832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203062.788894:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203062.788895:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203062.788899:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187784832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203062.788904:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187784960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203062.788922:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203062.788933:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203062/real 1788203062] req@ffff8e828793e300 x1875062187784960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203062.788946:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187784960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203062.788955:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203062.788957:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203062.788960:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187784960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203063.492496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203064.516467:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203065.540506:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203066.564476:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203067.588522:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203067.908456:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203067.908466:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203067.908471:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203067.908483:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4522 00000100:00080000:2.0:1788203067.908495:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203067.908503:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203067.908557:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187785088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203067.908587:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203067.908591:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203067.908593:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203067.908598:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4522 00000100:00080000:2.0:1788203067.908602:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203067.908609:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203067.908621:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203067.908645:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187785088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203067.908654:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187785216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203067.908666:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203067.908669:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203067.908670:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203067.908674:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4522 00000100:00080000:2.0:1788203067.908677:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203067.908680:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203067.908684:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203067.908695:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187785344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203067.908703:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203067.908705:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203067.908707:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203067.908709:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4522 00000100:00080000:2.0:1788203067.908711:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203067.908713:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203067.908718:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203067.908726:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187785472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203067.908733:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203067.908739:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203067.908742:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203067.908744:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203067.908748:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4522 00000100:00080000:2.0:1788203067.908751:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203067.908755:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203067.908758:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203067.908765:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187785600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203067.908787:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203067/real 1788203067] req@ffff8e82894e4e00 x1875062187785088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203067.908803:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187785088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203067.908813:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203067.908816:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203067.908819:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187785088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203067.908843:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187785216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203067.908859:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203067.908868:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203067/real 1788203067] req@ffff8e82894e4a80 x1875062187785216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203067.908877:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187785216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203067.908882:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203067.908885:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203067.908887:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187785216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203067.908890:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187785344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203067.908902:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203067.908908:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203067/real 1788203067] req@ffff8e82894e7800 x1875062187785344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203067.908916:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187785344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203067.908921:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203067.908923:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203067.908925:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187785344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203067.908928:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187785472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203067.908941:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203067.908947:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203067/real 1788203067] req@ffff8e82894e6300 x1875062187785472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203067.908954:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187785472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203067.908959:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203067.908961:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203067.908963:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187785472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203067.908966:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187785600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203067.908976:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203067.908982:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203067/real 1788203067] req@ffff8e82894e7100 x1875062187785600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203067.908994:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187785600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203067.908999:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203067.909001:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203067.909003:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187785600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203068.612526:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203069.636526:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203070.660538:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203071.684491:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203072.708578:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203073.028407:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203073.028415:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203073.028419:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203073.028427:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4528 00000100:00080000:2.0:1788203073.028436:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203073.028443:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203073.028475:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187785728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203073.028505:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203073.028508:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203073.028509:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203073.028512:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4528 00000100:00080000:2.0:1788203073.028514:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203073.028517:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203073.028535:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203073.028559:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187785856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203073.028569:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203073.028572:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203073.028574:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203073.028579:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4528 00000100:00080000:2.0:1788203073.028583:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203073.028588:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203073.028594:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203073.028603:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187785728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203073.028620:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187785984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203073.028639:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203073.028642:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203073.028644:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203073.028647:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4528 00000100:00080000:2.0:1788203073.028650:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203073.028654:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203073.028659:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203073.028669:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187786112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203073.028676:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203073.028679:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203073.028680:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203073.028683:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4528 00000100:00080000:2.0:1788203073.028685:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203073.028688:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203073.028690:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203073.028699:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187786240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203073.028924:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203073.029022:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203073/real 1788203073] req@ffff8e82894e6d80 x1875062187785728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203073.029039:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187785728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203073.029049:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203073.029053:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203073.029057:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187785728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203073.029065:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187785856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203073.029085:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203073.029094:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203073/real 1788203073] req@ffff8e82894e5f80 x1875062187785856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203073.029103:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187785856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203073.029109:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203073.029111:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203073.029114:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187785856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203073.029136:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187785984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203073.029148:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203073.029155:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203073/real 1788203073] req@ffff8e82894e4380 x1875062187785984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203073.029163:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187785984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203073.029168:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203073.029170:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203073.029172:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187785984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203073.029176:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187786112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203073.029188:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203073.029195:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203073/real 1788203073] req@ffff8e82894e5c00 x1875062187786112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203073.029202:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187786112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203073.029208:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203073.029209:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203073.029211:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187786112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203073.029215:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187786240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203073.029226:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203073.029232:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203073/real 1788203073] req@ffff8e82894e7100 x1875062187786240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203073.029239:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187786240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203098 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203073.029244:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203073.029246:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203073.029248:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187786240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203073.732564:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203074.756541:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203075.780533:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203076.804539:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203077.828501:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203078.148447:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203078.148477:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203078.148482:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203078.148496:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4533 00000100:00080000:2.0:1788203078.148511:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203078.148519:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203078.148577:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187786368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203078.148616:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203078.148620:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203078.148622:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203078.148626:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4533 00000100:00080000:2.0:1788203078.148630:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203078.148635:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203078.148646:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203078.148662:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187786496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203078.148667:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187786368:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203078.148683:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203078.148690:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203078.148692:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203078.148698:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4533 00000100:00080000:2.0:1788203078.148702:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203078.148706:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203078.148713:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203078.148725:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187786624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203078.148735:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203078.148738:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203078.148740:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788203078.148745:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203078.148747:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4533 00000100:00080000:2.0:1788203078.148753:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203078.148758:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203078.148764:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203078.148776:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187786752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203078.148786:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203078.148790:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203078.148791:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203078.148796:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4533 00000100:00080000:2.0:1788203078.148800:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203078.148805:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203078.148811:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203078/real 1788203078] req@ffff8e82894e6300 x1875062187786368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788203078.148814:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203078.148829:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187786880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203078.148840:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187786368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203078.148856:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203078.148861:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203078.148866:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187786368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203078.148893:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187786496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203078.148919:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203078.148933:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203078/real 1788203078] req@ffff8e82894e7800 x1875062187786496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203078.148946:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187786496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203078.148962:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203078.148966:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203078.148970:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187786496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203078.148977:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187786624:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203078.148995:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203078.149004:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203078/real 1788203078] req@ffff8e82894e4a80 x1875062187786624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203078.149012:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187786624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203078.149018:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203078.149020:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203078.149022:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187786624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203078.149026:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187786752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203078.149044:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203078.149051:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203078/real 1788203078] req@ffff8e82894e4e00 x1875062187786752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203078.149058:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187786752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203078.149064:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203078.149066:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203078.149069:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187786752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203078.149073:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187786880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203078.149085:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203078.149090:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203078/real 1788203078] req@ffff8e82894e4000 x1875062187786880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203078.149097:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187786880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203103 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203078.149102:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203078.149104:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203078.149106:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187786880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203078.852553:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203079.876491:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203080.900504:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203081.924470:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203082.948502:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203083.268406:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203083.268413:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203083.268417:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203083.268426:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4538 00000100:00080000:0.0:1788203083.268435:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203083.268440:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203083.268472:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187787008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203083.268499:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203083.268502:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203083.268503:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203083.268507:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4538 00000100:00080000:0.0:1788203083.268511:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203083.268515:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203083.268525:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203083.268540:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187787136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203083.268550:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203083.268554:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203083.268557:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203083.268561:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4538 00000100:00080000:0.0:1788203083.268565:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788203083.268570:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187787008:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203083.268573:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203083.268584:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203083.268594:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187787264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203083.268602:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203083.268605:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203083.268607:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203083.268609:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4538 00000100:00080000:0.0:1788203083.268612:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203083.268615:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203083.268618:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203083.268625:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187787392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203083.268631:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203083.268634:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203083.268636:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203083.268638:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4538 00000100:00080000:0.0:1788203083.268641:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203083.268645:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:3.0:1788203083.268648:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788203083.268651:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203083.268666:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187787520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203083.268697:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203083/real 1788203083] req@ffff8e828793d500 x1875062187787008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203083.268724:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187787008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203083.268732:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203083.268735:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203083.268738:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187787008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203083.268744:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187787136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203083.268762:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203083.268770:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203083/real 1788203083] req@ffff8e828793d880 x1875062187787136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203083.268778:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187787136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203083.268783:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203083.268785:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203083.268787:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187787136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203083.268804:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187787264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203083.268815:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203083.268821:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203083/real 1788203083] req@ffff8e828793fb80 x1875062187787264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203083.268828:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187787264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203083.268838:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203083.268840:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203083.268842:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187787264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203083.268845:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187787392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203083.268855:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203083.268861:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203083/real 1788203083] req@ffff8e828793c380 x1875062187787392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203083.268867:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187787392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203083.268873:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203083.268874:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203083.268876:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187787392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203083.268879:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187787520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203083.268889:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203083.268899:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203083/real 1788203083] req@ffff8e828793e300 x1875062187787520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203083.268910:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187787520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203108 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203083.268919:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203083.268922:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203083.268927:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187787520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203083.972460:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203084.996569:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203086.020501:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203087.044456:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203088.068454:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203088.388437:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203088.388450:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203088.388455:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203088.388466:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4543 00000100:00080000:0.0:1788203088.388479:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203088.388487:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203088.388543:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187787648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203088.388572:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203088.388575:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203088.388577:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203088.388581:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4543 00000100:00080000:0.0:1788203088.388583:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203088.388586:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203088.388595:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203088.388604:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187787776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203088.388614:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203088.388616:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203088.388618:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203088.388623:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4543 00000100:00100000:3.0:1788203088.388623:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187787648:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203088.388634:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203088.388639:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203088.388644:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203088.388653:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187787904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203088.388660:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203088.388663:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203088.388665:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203088.388667:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4543 00000100:00080000:0.0:1788203088.388669:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203088.388672:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203088.388676:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203088.388682:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187788032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203088.388689:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203088.388691:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203088.388693:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203088.388695:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4543 00000100:00080000:0.0:1788203088.388701:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203088.388704:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203088.388707:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788203088.388715:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203088.388718:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187788160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203088.388771:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203088/real 1788203088] req@ffff8e828793f480 x1875062187787648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203088.388787:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187787648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203088.388797:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203088.388801:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203088.388804:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187787648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203088.388827:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187787776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203088.388852:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203088.388862:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203088/real 1788203088] req@ffff8e828793dc00 x1875062187787776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203088.388870:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187787776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203088.388876:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203088.388879:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203088.388881:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187787776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203088.388885:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187787904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203088.388896:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203088.388904:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203088/real 1788203088] req@ffff8e828793c000 x1875062187787904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203088.388911:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187787904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203088.388917:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203088.388918:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203088.388920:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187787904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203088.388924:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187788032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203088.388936:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203088.388942:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203088/real 1788203088] req@ffff8e828793ca80 x1875062187788032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203088.388950:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187788032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203088.388955:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203088.388956:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203088.388958:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187788032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203088.388961:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187788160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203088.388972:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203088.388977:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203088/real 1788203088] req@ffff8e828793c700 x1875062187788160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203088.388983:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187788160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203088.388989:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203088.388991:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203088.388993:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187788160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203089.092454:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203090.116481:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203091.140471:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203092.164472:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203093.188488:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203093.508427:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203093.508434:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203093.508438:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203093.508447:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4548 00000100:00080000:0.0:1788203093.508456:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203093.508463:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203093.508496:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187788288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203093.508516:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203093.508519:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203093.508521:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203093.508523:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4548 00000100:00080000:0.0:1788203093.508526:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203093.508528:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203093.508535:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203093.508544:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187788416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203093.508552:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203093.508557:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203093.508559:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203093.508561:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187788288:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203093.508563:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4548 00000100:00080000:0.0:1788203093.508569:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203093.508572:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203093.508575:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203093.508589:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187788544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203093.508598:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203093.508607:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203093.508609:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203093.508614:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4548 00000100:00080000:0.0:1788203093.508625:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203093.508629:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:3.0:1788203093.508635:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788203093.508638:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203093.508653:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187788672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203093.508661:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203093.508665:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203093.508667:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203093.508671:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4548 00000100:00080000:0.0:1788203093.508676:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788203093.508680:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203093/real 1788203093] req@ffff8e828793ce00 x1875062187788288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203093.508684:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203093.508695:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203093.508702:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187788288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203093.508712:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187788800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203093.508713:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203093.508721:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203093.508728:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187788288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203093.508750:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187788416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203093.508767:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203093.508775:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203093/real 1788203093] req@ffff8e828793f800 x1875062187788416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203093.508783:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187788416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203093.508788:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203093.508790:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203093.508792:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187788416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203093.508796:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187788544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203093.508807:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203093.508813:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203093/real 1788203093] req@ffff8e828793e680 x1875062187788544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203093.508820:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187788544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203093.508825:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203093.508826:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203093.508828:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187788544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203093.508831:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187788672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203093.508841:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203093.508847:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203093/real 1788203093] req@ffff8e828793f100 x1875062187788672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203093.508854:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187788672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203093.508858:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203093.508860:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203093.508862:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187788672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203093.508865:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187788800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203093.508874:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203093.508879:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203093/real 1788203093] req@ffff8e828793c700 x1875062187788800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203093.508886:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187788800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203093.508891:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203093.508893:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203093.508894:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187788800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203094.212526:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203095.236499:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203096.260555:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203097.284520:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203098.308514:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203098.628437:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203098.628446:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203098.628450:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203098.628458:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4553 00000100:00080000:0.0:1788203098.628468:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203098.628476:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203098.628526:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187788928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203098.628554:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203098.628557:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203098.628559:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203098.628563:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4553 00000100:00080000:0.0:1788203098.628566:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203098.628570:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203098.628578:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203098.628589:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187788928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203098.628592:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187789056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203098.628603:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203098.628605:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203098.628607:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203098.628610:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4553 00000100:00080000:0.0:1788203098.628613:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203098.628616:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203098.628625:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203098.628634:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187789184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203098.628642:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203098.628646:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1788203098.628650:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203098.628651:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203098.628659:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4553 00000100:00080000:0.0:1788203098.628664:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203098.628670:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203098.628676:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203098.628690:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203098/real 1788203098] req@ffff8e828793ca80 x1875062187788928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203098.628694:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187789312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203098.628711:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788203098.628715:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187788928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203098.628716:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203098.628720:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203098.628728:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4553 00000100:00080000:0.0:1788203098.628734:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203098.628734:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203098.628743:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:0.0:1788203098.628744:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203098.628750:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187788928:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788203098.628751:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203098.628765:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187789440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203098.628783:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187789056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203098.628811:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203098.628826:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203098/real 1788203098] req@ffff8e828793c000 x1875062187789056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203098.628850:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187789056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203098.628867:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203098.628871:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203098.628876:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187789056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203098.628883:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187789184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203098.628906:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203098.628919:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203098/real 1788203098] req@ffff8e828793dc00 x1875062187789184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203098.628941:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187789184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203098.628948:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203098.628950:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203098.628953:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187789184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203098.628957:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187789312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203098.628974:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203098.628981:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203098/real 1788203098] req@ffff8e828793f480 x1875062187789312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203098.628990:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187789312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203098.628998:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203098.629001:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203098.629005:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187789312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203098.629023:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187789440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203098.629044:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203098.629056:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203098/real 1788203098] req@ffff8e828793e300 x1875062187789440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203098.629067:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187789440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203098.629072:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203098.629074:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203098.629076:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187789440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203099.332489:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203100.356492:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203101.380522:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203102.404564:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203103.428501:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203103.748395:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203103.748402:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203103.748406:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203103.748414:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4558 00000100:00080000:0.0:1788203103.748424:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203103.748429:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203103.748463:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187789568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203103.748493:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203103.748497:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203103.748499:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203103.748510:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4558 00000100:00080000:0.0:1788203103.748514:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203103.748518:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203103.748529:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203103.748542:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187789568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203103.748543:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187789696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203103.748558:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203103.748560:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203103.748561:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203103.748565:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4558 00000100:00080000:0.0:1788203103.748567:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203103.748569:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203103.748574:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203103.748586:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187789824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203103.748603:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203103.748607:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203103.748609:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203103.748613:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4558 00000100:00080000:0.0:1788203103.748618:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203103.748628:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203103.748634:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203103.748644:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187789696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203103.748648:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187789952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203103.748663:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203103.748667:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203103.748670:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203103.748674:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187789824:192.168.201.121@tcp:38:kworker.0 00000100:00080000:0.0:1788203103.748679:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4558 00000100:00080000:0.0:1788203103.748686:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203103.748692:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203103.748699:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203103.748699:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187789952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203103.748715:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187790080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203103.748734:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187790080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203104.452496:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203105.476504:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203106.500509:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203106.820648:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788203106.820694:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788203106.820712:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203106.820753:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203106.820763:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203106.820773:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203106.820783:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:0.0:1788203106.820806:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203103/real 1788203106] req@ffff8e828793e300 x1875062187790080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203106.820845:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187790080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203106.820858:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203106.820864:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203106.820868:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187790080:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203106.820877:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203103/real 1788203106] req@ffff8e828793d500 x1875062187789952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203106.820890:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187789952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203106.820897:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203106.820900:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203106.820904:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187789952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203106.820911:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203103/real 1788203106] req@ffff8e828793fb80 x1875062187789696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203106.820923:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187789696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203106.820937:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203106.820942:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203106.820945:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187789696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203106.820952:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203103/real 1788203106] req@ffff8e828793d880 x1875062187789824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203106.820979:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187789824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203106.820988:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203106.820991:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203106.820995:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187789824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203106.821003:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203103/real 1788203106] req@ffff8e828793c380 x1875062187789568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203106.821015:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187789568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203106.821022:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203106.821025:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203106.821029:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187789568:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788203107.524482:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203108.548608:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203108.868421:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203108.868432:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203108.868437:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203108.868449:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4563 00000100:00080000:0.0:1788203108.868461:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203108.868469:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203108.868505:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187790208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203108.868535:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203108.868542:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203108.868544:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203108.868550:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4563 00000100:00080000:0.0:1788203108.868555:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203108.868562:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203108.868562:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187790208:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788203108.868576:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203108.868595:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187790336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203108.868607:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203108.868611:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203108.868613:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203108.868617:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4563 00000100:00080000:0.0:1788203108.868622:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203108.868627:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203108.868634:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788203108.868635:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203108.868652:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187790464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203108.868664:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203108.868668:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203108.868671:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203108.868676:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4563 00000100:00080000:0.0:1788203108.868680:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203108.868684:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203108.868690:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203108.868703:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203108/real 1788203108] req@ffff8e828793c380 x1875062187790208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203108.868706:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187790592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203108.868726:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203108.868732:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203108.868734:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203108.868739:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4563 00000100:00080000:0.0:1788203108.868743:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203108.868744:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187790208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203108.868753:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203108.868763:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788203108.868765:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203108.868773:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203108.868780:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187790720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203108.868781:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187790208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203108.868813:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187790336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203108.868840:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203108.868853:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203108/real 1788203108] req@ffff8e828793d880 x1875062187790336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203108.868866:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187790336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203108.868874:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203108.868878:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203108.868882:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187790336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203108.868888:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187790464:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203108.868907:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203108.868917:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203108/real 1788203108] req@ffff8e828793fb80 x1875062187790464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203108.868946:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187790464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203108.868956:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203108.868959:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203108.868963:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187790464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203108.868969:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187790592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203108.868990:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203108.869001:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203108/real 1788203108] req@ffff8e828793d500 x1875062187790592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203108.869013:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187790592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203108.869022:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203108.869025:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203108.869028:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187790592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203108.869035:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187790720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203108.869054:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203108.869065:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203108/real 1788203108] req@ffff8e828793e300 x1875062187790720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203108.869082:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187790720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203108.869091:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203108.869094:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203108.869097:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187790720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203109.572467:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203110.596483:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203111.620429:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203112.644478:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203113.668450:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203113.988411:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203113.988419:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203113.988423:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203113.988431:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4569 00000100:00080000:2.0:1788203113.988439:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203113.988445:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203113.988479:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187790848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203113.988509:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203113.988514:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203113.988516:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203113.988522:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4569 00000100:00080000:2.0:1788203113.988526:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203113.988531:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203113.988542:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203113.988547:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187790848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203113.988560:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187790976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203113.988580:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203113.988583:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203113.988585:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203113.988589:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4569 00000100:00080000:2.0:1788203113.988594:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203113.988598:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203113.988603:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788203113.988616:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203113.988618:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187791104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203113.988630:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203113.988634:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203113.988637:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203113.988641:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4569 00000100:00080000:2.0:1788203113.988645:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203113.988650:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203113.988656:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203113.988659:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203113/real 1788203113] req@ffff8e82894e6a00 x1875062187790848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203113.988670:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187791232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203113.988676:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187790848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203113.988693:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203113.988696:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203113.988699:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203113.988702:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:2.0:1788203113.988704:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788203113.988708:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187790848:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203113.988713:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4569 00000100:00080000:2.0:1788203113.988719:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203113.988723:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203113.988729:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203113.988732:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187790976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203113.988748:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187791360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203113.988757:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203113.988773:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203113/real 1788203113] req@ffff8e82894e7480 x1875062187790976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203113.988782:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187790976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203113.988788:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203113.988791:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203113.988793:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187790976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203113.988797:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187791104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203113.988809:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203113.988814:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203113/real 1788203113] req@ffff8e82894e6680 x1875062187791104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203113.988822:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187791104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203113.988826:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203113.988828:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203113.988830:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187791104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203113.988859:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187791232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203113.988874:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203113.988882:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203113/real 1788203113] req@ffff8e82894e5880 x1875062187791232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203113.988894:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187791232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203113.988900:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203113.988902:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203113.988904:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187791232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203113.988913:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187791360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203113.988924:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203113.988930:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203113/real 1788203113] req@ffff8e82894e6a00 x1875062187791360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203113.988937:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187791360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203113.988943:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203113.988944:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203113.988946:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187791360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203114.692448:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203115.716550:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203116.740486:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203117.764510:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203118.788492:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203119.108448:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203119.108464:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203119.108470:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203119.108484:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4574 00000100:00080000:0.0:1788203119.108500:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203119.108510:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203119.108559:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187791488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203119.108591:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203119.108595:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203119.108597:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203119.108603:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4574 00000100:00080000:0.0:1788203119.108607:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203119.108612:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203119.108624:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203119.108629:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187791488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203119.108641:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187791616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203119.108653:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203119.108658:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203119.108660:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203119.108664:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4574 00000100:00080000:0.0:1788203119.108668:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203119.108673:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203119.108679:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203119.108691:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187791744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203119.108701:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788203119.108706:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203119.108710:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203119.108714:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203119.108720:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4574 00000100:00080000:0.0:1788203119.108726:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203119.108732:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203119.108738:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203119.108765:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187791872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203119.108778:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788203119.108778:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203119/real 1788203119] req@ffff8e828793f480 x1875062187791488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203119.108788:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203119.108792:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203119.108801:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4574 00000100:00080000:0.0:1788203119.108805:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203119.108810:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187791488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203119.108812:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203119.108819:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788203119.108831:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788203119.108834:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187792000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203119.108837:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203119.108845:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187791488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203119.108885:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187791616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203119.108905:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203119.108919:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203119/real 1788203119] req@ffff8e828793dc00 x1875062187791616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203119.108933:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187791616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203119.108943:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203119.108947:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203119.108951:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187791616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203119.108958:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187791744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203119.108978:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203119.108991:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203119/real 1788203119] req@ffff8e828793c000 x1875062187791744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203119.109004:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187791744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203119.109013:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203119.109016:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203119.109021:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187791744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203119.109028:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187791872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203119.109049:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203119.109063:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203119/real 1788203119] req@ffff8e828793ca80 x1875062187791872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203119.109089:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187791872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203119.109099:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203119.109102:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203119.109105:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187791872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203119.109112:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187792000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203119.109133:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203119.109143:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203119/real 1788203119] req@ffff8e828793c700 x1875062187792000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203144 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203119.109151:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187792000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203144 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203119.109157:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203119.109160:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203119.109162:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187792000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203119.812498:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203120.836522:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203121.860565:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203122.884464:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203123.908482:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203124.228376:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203124.228386:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203124.228390:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203124.228402:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4579 00000100:00080000:0.0:1788203124.228413:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203124.228420:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203124.228450:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187792128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203124.228471:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203124.228477:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203124.228479:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203124.228484:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4579 00000100:00080000:0.0:1788203124.228488:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203124.228492:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203124.228501:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203124.228508:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187792128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203124.228514:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187792256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203124.228523:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203124.228526:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203124.228528:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203124.228532:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4579 00000100:00080000:0.0:1788203124.228535:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203124.228538:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203124.228542:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203124.228551:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187792384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203124.228558:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203124.228561:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203124.228563:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203124.228566:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4579 00000100:00080000:0.0:1788203124.228569:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203124.228574:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:2.0:1788203124.228574:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788203124.228579:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203124.228589:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187792512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203124.228597:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203124.228600:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203124.228602:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203124.228605:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4579 00000100:00080000:0.0:1788203124.228608:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203124.228611:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203124.228615:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203124.228621:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203124/real 1788203124] req@ffff8e828793f100 x1875062187792128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203124.228624:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187792640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203124.228639:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187792128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203124.228650:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203124.228669:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203124.228673:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187792128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203124.228697:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187792256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203124.228722:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203124.228736:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203124/real 1788203124] req@ffff8e828793e680 x1875062187792256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203124.228747:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187792256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203124.228758:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203124.228760:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203124.228762:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187792256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203124.228766:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187792384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203124.228790:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203124.228797:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203124/real 1788203124] req@ffff8e828793f800 x1875062187792384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203124.228805:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187792384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203124.228810:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203124.228812:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203124.228814:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187792384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203124.228817:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187792512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203124.228828:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203124.228834:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203124/real 1788203124] req@ffff8e828793ce00 x1875062187792512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203124.228840:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187792512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203124.228845:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203124.228847:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203124.228849:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187792512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203124.228853:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187792640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203124.228862:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203124.228867:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203124/real 1788203124] req@ffff8e828793c700 x1875062187792640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203149 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203124.228874:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187792640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203149 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203124.228879:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203124.228881:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203124.228882:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187792640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203124.932470:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203125.956481:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203126.980489:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203128.004509:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203129.028499:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203129.348403:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203129.348412:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203129.348417:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203129.348428:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4584 00000100:00080000:0.0:1788203129.348438:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203129.348446:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203129.348482:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187792768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203129.348511:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203129.348516:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203129.348517:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203129.348521:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4584 00000100:00080000:0.0:1788203129.348526:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203129.348530:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203129.348539:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203129.348551:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187792896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203129.348562:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203129.348577:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203129.348579:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788203129.348585:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187792768:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203129.348586:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4584 00000100:00080000:0.0:1788203129.348593:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203129.348598:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203129.348604:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203129.348616:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187793024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203129.348627:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203129.348631:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203129.348634:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203129.348637:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4584 00000100:00080000:0.0:1788203129.348648:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203129.348651:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:2.0:1788203129.348657:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788203129.348658:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203129.348671:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187793152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203129.348688:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203129.348692:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203129.348694:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203129.348699:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4584 00000100:00080000:0.0:1788203129.348703:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203129.348708:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203129/real 1788203129] req@ffff8e828793ca80 x1875062187792768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203129.348709:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203129.348716:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203129.348731:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187793280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203129.348740:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187792768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203129.348751:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203129.348755:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203129.348758:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187792768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203129.348765:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187792896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203129.348786:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203129.348796:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203129/real 1788203129] req@ffff8e828793c000 x1875062187792896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203129.348804:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187792896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203129.348809:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203129.348811:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203129.348813:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187792896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203129.348843:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187793024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203129.348857:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203129.348864:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203129/real 1788203129] req@ffff8e828793dc00 x1875062187793024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203129.348871:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187793024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203129.348876:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203129.348878:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203129.348880:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187793024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203129.348884:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187793152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203129.348894:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203129.348900:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203129/real 1788203129] req@ffff8e828793f480 x1875062187793152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203129.348907:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187793152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203129.348912:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203129.348914:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203129.348916:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187793152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203129.348919:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187793280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203129.348929:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203129.348934:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203129/real 1788203129] req@ffff8e828793e300 x1875062187793280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203129.348941:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187793280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203129.348946:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203129.348948:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203129.348949:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187793280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203130.052469:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203131.076480:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203132.100541:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203133.124489:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203134.148445:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203134.468500:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203134.468513:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203134.468516:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203134.468525:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4589 00000100:00080000:1.0:1788203134.468536:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203134.468543:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203134.468582:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187793408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203134.468612:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203134.468616:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203134.468618:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203134.468622:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4589 00000100:00080000:1.0:1788203134.468625:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203134.468628:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203134.468636:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203134.468646:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187793536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203134.468654:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788203134.468658:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187793408:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788203134.468659:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203134.468663:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203134.468668:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4589 00000100:00080000:1.0:1788203134.468672:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203134.468674:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203134.468678:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203134.468686:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187793664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203134.468692:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203134.468695:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203134.468696:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203134.468699:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4589 00000100:00080000:1.0:1788203134.468702:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203134.468705:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203134.468708:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203134.468714:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187793792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203134.468720:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203134.468722:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203134.468723:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203134.468726:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4589 00000100:00080000:1.0:1788203134.468729:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788203134.468733:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788203134.468736:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203134.468740:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203134.468748:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187793920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203134.468780:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203134/real 1788203134] req@ffff8e828d720380 x1875062187793408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203134.468795:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187793408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203134.468803:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203134.468807:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203134.468811:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187793408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203134.468837:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187793536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203134.468862:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203134.468873:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203134/real 1788203134] req@ffff8e828d722300 x1875062187793536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203134.468881:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187793536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203134.468887:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203134.468889:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203134.468891:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187793536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203134.468895:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187793664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203134.468907:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203134.468913:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203134/real 1788203134] req@ffff8e828d722d80 x1875062187793664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203134.468921:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187793664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203134.468926:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203134.468927:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203134.468929:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187793664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203134.468932:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187793792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203134.468942:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203134.468948:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203134/real 1788203134] req@ffff8e828d721180 x1875062187793792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203134.468956:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187793792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203134.468961:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203134.468962:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203134.468964:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187793792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203134.468968:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187793920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203134.468977:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203134.468982:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203134/real 1788203134] req@ffff8e828d720a80 x1875062187793920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203134.468988:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187793920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203134.468993:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203134.468994:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203134.468996:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187793920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203135.172491:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203136.196426:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203137.220511:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203138.244484:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203139.268477:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203139.588369:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203139.588377:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203139.588381:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203139.588401:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4594 00000100:00080000:1.0:1788203139.588411:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203139.588417:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203139.588452:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187794048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203139.588476:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203139.588479:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203139.588481:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203139.588488:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4594 00000100:00080000:1.0:1788203139.588490:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203139.588493:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203139.588500:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203139.588509:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187794176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203139.588518:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203139.588522:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203139.588524:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203139.588529:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4594 00000100:00100000:2.0:1788203139.588534:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187794048:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788203139.588536:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203139.588544:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203139.588550:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203139.588566:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187794304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203139.588577:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203139.588579:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203139.588581:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203139.588585:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4594 00000100:00080000:1.0:1788203139.588588:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203139.588591:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203139.588595:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203139.588604:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187794432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203139.588607:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788203139.588616:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203139.588619:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203139.588620:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203139.588624:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4594 00000100:00080000:1.0:1788203139.588628:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203139.588631:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203139.588633:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203139.588641:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187794560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203139.588649:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203139/real 1788203139] req@ffff8e828d720a80 x1875062187794048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203139.588665:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187794048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203139.588678:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203139.588680:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203139.588683:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187794048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203139.588688:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187794176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203139.588702:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203139.588710:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203139/real 1788203139] req@ffff8e828d721180 x1875062187794176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203139.588718:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187794176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203139.588723:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203139.588725:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203139.588727:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187794176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203139.588746:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187794304:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203139.588757:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203139.588763:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203139/real 1788203139] req@ffff8e828d722d80 x1875062187794304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203139.588770:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187794304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203139.588775:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203139.588777:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203139.588778:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187794304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203139.588783:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187794432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203139.588793:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203139.588798:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203139/real 1788203139] req@ffff8e828d722300 x1875062187794432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203139.588806:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187794432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203139.588810:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203139.588812:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203139.588814:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187794432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203139.588817:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187794560:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203139.588827:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203139.588832:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203139/real 1788203139] req@ffff8e828d720380 x1875062187794560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203139.588839:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187794560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203139.588844:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203139.588845:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203139.588847:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187794560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203140.292458:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203141.316517:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203142.340477:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203143.364502:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203144.388497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203144.708418:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203144.708428:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203144.708433:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203144.708443:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4599 00000100:00080000:1.0:1788203144.708455:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203144.708464:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203144.708503:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187794688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203144.708534:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203144.708539:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203144.708541:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203144.708546:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4599 00000100:00080000:1.0:1788203144.708550:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203144.708555:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203144.708565:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203144.708583:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187794688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203144.708585:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187794816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203144.708606:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203144.708611:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203144.708614:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203144.708619:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4599 00000100:00080000:1.0:1788203144.708625:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203144.708630:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203144.708637:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203144.708661:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187794944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203144.708670:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788203144.708681:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203144.708685:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203144.708688:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203144.708694:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4599 00000100:00080000:1.0:1788203144.708699:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203144.708704:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203144.708711:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203144.708717:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203144/real 1788203144] req@ffff8e828d720e00 x1875062187794688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203144.708729:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187795072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203144.708742:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788203144.708743:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187794688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203144.708753:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203144.708758:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203144.708760:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203144.708766:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4599 00000100:00080000:2.0:1788203144.708767:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:1.0:1788203144.708775:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203144.708775:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187794688:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788203144.708786:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203144.708792:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203144.708803:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187794816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203144.708810:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187795200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203144.708835:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203144.708852:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203144/real 1788203144] req@ffff8e828d723800 x1875062187794816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203144.708867:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187794816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203144.708876:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203144.708880:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203144.708884:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187794816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203144.708889:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187794944:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203144.708907:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203144.708918:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203144/real 1788203144] req@ffff8e828d721500 x1875062187794944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203144.708937:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187794944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203144.708952:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203144.708956:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203144.708959:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187794944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203144.708967:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187795072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203144.708993:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203144.709010:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203144/real 1788203144] req@ffff8e828d720000 x1875062187795072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203144.709023:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187795072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203144.709034:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203144.709038:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203144.709043:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187795072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203144.709072:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187795200:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203144.709096:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203144.709106:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203144/real 1788203144] req@ffff8e828d722680 x1875062187795200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203144.709113:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187795200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203144.709121:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203144.709124:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203144.709128:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187795200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203145.412490:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203146.436495:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203147.460475:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203148.484484:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203149.508482:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203149.828442:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203149.828451:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203149.828455:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203149.828463:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4604 00000100:00080000:2.0:1788203149.828472:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203149.828478:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203149.828511:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187795328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203149.828535:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203149.828538:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203149.828539:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203149.828542:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4604 00000100:00080000:2.0:1788203149.828545:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203149.828548:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203149.828556:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203149.828567:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187795328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203149.828568:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187795456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203149.828581:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203149.828584:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203149.828585:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203149.828587:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4604 00000100:00080000:2.0:1788203149.828589:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203149.828592:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203149.828595:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203149.828609:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187795584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203149.828615:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203149.828618:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203149.828620:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203149.828624:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4604 00000100:00080000:2.0:1788203149.828628:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788203149.828630:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203149.828636:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203149.828642:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203149.828656:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187795712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203149.828667:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203149.828671:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203149.828673:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203149.828678:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4604 00000100:00100000:0.0:1788203149.828682:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203149/real 1788203149] req@ffff8e82894e4000 x1875062187795328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203149.828684:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203149.828693:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203149.828701:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203149.828704:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187795328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203149.828714:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203149.828719:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203149.828719:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187795840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203149.828727:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187795328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203149.828753:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187795456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203149.828770:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203149.828779:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203149/real 1788203149] req@ffff8e82894e4e00 x1875062187795456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203149.828787:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187795456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203149.828792:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203149.828795:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203149.828798:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187795456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203149.828805:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187795584:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203149.828824:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203149.828837:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203149/real 1788203149] req@ffff8e82894e4a80 x1875062187795584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203149.828849:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187795584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203149.828858:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203149.828860:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203149.828862:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187795584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203149.828867:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187795712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203149.828884:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203149.828892:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203149/real 1788203149] req@ffff8e82894e7800 x1875062187795712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203149.828900:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187795712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203149.828905:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203149.828908:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203149.828911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187795712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203149.828917:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187795840:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203149.828937:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203149.828949:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203149/real 1788203149] req@ffff8e82894e6300 x1875062187795840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203149.828962:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187795840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203149.828970:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203149.828974:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203149.828978:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187795840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203150.532475:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203151.556465:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203152.580474:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203153.604484:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203154.628493:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203154.948408:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203154.948417:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203154.948421:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203154.948429:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4610 00000100:00080000:1.0:1788203154.948439:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203154.948445:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203154.948481:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187795968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203154.948506:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203154.948511:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203154.948513:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203154.948517:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4610 00000100:00080000:1.0:1788203154.948521:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203154.948526:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203154.948535:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203154.948551:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187796096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203154.948562:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203154.948565:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203154.948567:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203154.948569:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4610 00000100:00080000:1.0:1788203154.948573:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203154.948575:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203154.948579:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203154.948584:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187795968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203154.948591:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187796224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203154.948599:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203154.948601:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203154.948602:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203154.948606:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4610 00000100:00080000:1.0:1788203154.948608:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203154.948611:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203154.948614:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203154.948622:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187796352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203154.948628:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203154.948630:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203154.948631:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203154.948634:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4610 00000100:00080000:1.0:1788203154.948636:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203154.948638:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203154.948641:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788203154.948646:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203154.948650:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187796480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203154.948688:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203154/real 1788203154] req@ffff8e828d722680 x1875062187795968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203154.948702:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187795968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203154.948710:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203154.948713:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203154.948716:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187795968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203154.948721:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187796096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203154.948735:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203154.948743:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203154/real 1788203154] req@ffff8e828d720000 x1875062187796096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203154.948750:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187796096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203154.948756:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203154.948758:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203154.948760:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187796096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203154.948778:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187796224:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203154.948789:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203154.948795:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203154/real 1788203154] req@ffff8e828d721500 x1875062187796224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203154.948802:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187796224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203154.948811:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203154.948813:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203154.948815:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187796224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203154.948819:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187796352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203154.948829:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203154.948834:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203154/real 1788203154] req@ffff8e828d723800 x1875062187796352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203154.948842:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187796352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203154.948850:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203154.948853:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203154.948857:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187796352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203154.948862:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187796480:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203154.948882:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203154.948893:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203154/real 1788203154] req@ffff8e828d720e00 x1875062187796480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203154.948906:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187796480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203154.948914:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203154.948916:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203154.948919:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187796480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203155.652455:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203156.676557:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203157.700449:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203158.724547:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203159.748489:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203160.068439:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203160.068451:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203160.068456:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203160.068468:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4615 00000100:00080000:1.0:1788203160.068481:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203160.068488:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203160.068534:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187796608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203160.068562:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203160.068565:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203160.068566:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203160.068570:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4615 00000100:00080000:1.0:1788203160.068573:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203160.068575:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203160.068586:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203160.068596:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187796736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203160.068604:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203160.068607:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203160.068608:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203160.068611:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4615 00000100:00080000:1.0:1788203160.068613:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203160.068616:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203160.068619:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203160.068628:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187796864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203160.068636:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187796608:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788203160.068639:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203160.068650:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203160.068655:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203160.068661:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4615 00000100:00080000:1.0:1788203160.068665:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203160.068668:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203160.068673:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203160.068689:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187796992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203160.068695:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203160.068698:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203160.068699:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203160.068702:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4615 00000100:00080000:1.0:1788203160.068705:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203160.068707:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203160.068712:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203160.068719:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187797120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203160.068743:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203160.068804:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203160/real 1788203160] req@ffff8e828d720380 x1875062187796608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203160.068828:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187796608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203160.068840:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203160.068844:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203160.068849:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187796608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203160.068858:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187796736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203160.068883:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203160.068898:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203160/real 1788203160] req@ffff8e828d722300 x1875062187796736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203160.068912:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187796736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203160.068923:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203160.068926:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203160.068930:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187796736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203160.068960:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187796864:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203160.068978:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203160.068988:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203160/real 1788203160] req@ffff8e828d722d80 x1875062187796864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203160.069000:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187796864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203160.069008:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203160.069012:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203160.069015:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187796864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203160.069021:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187796992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203160.069056:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203160.069067:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203160/real 1788203160] req@ffff8e828d721180 x1875062187796992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203160.069075:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187796992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203160.069081:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203160.069083:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203160.069085:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187796992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203160.069089:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187797120:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203160.069102:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203160.069108:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203160/real 1788203160] req@ffff8e828d720a80 x1875062187797120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203160.069116:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187797120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203185 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203160.069121:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203160.069123:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203160.069124:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187797120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203160.772496:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203161.796490:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203162.820553:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203163.844513:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203164.868526:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203165.188472:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203165.188484:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203165.188489:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203165.188501:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4620 00000100:00080000:1.0:1788203165.188514:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203165.188522:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203165.188582:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187797248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203165.188617:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203165.188620:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203165.188622:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203165.188625:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4620 00000100:00080000:1.0:1788203165.188628:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203165.188631:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203165.188640:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203165.188649:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187797376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203165.188656:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203165.188660:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203165.188661:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203165.188664:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4620 00000100:00080000:1.0:1788203165.188666:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203165.188669:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203165.188672:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203165.188678:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187797504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203165.188685:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203165.188688:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203165.188690:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788203165.188695:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187797248:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788203165.188695:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4620 00000100:00080000:1.0:1788203165.188698:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203165.188704:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203165.188710:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203165.188717:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187797632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203165.188722:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203165.188724:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203165.188725:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203165.188728:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4620 00000100:00080000:1.0:1788203165.188730:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203165.188732:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203165.188735:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203165.188742:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187797760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203165.188792:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203165.188840:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203165/real 1788203165] req@ffff8e828d720a80 x1875062187797248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203165.188856:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187797248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203165.188865:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203165.188868:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203165.188872:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187797248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203165.188878:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187797376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203165.188894:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203165.188902:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203165/real 1788203165] req@ffff8e828d721180 x1875062187797376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203165.188916:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187797376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203165.188921:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203165.188923:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203165.188925:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187797376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203165.188968:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187797504:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203165.188985:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203165.188993:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203165/real 1788203165] req@ffff8e828d722d80 x1875062187797504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203165.189001:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187797504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203165.189007:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203165.189009:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203165.189012:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187797504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203165.189016:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187797632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203165.189028:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203165.189034:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203165/real 1788203165] req@ffff8e828d722300 x1875062187797632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203165.189041:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187797632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203165.189052:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203165.189054:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203165.189056:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187797632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203165.189059:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187797760:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203165.189069:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203165.189075:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203165/real 1788203165] req@ffff8e828d720380 x1875062187797760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203190 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203165.189081:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187797760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203190 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203165.189086:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203165.189088:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203165.189091:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187797760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203165.892482:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203166.916474:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203167.940544:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203168.964508:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203169.988527:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203170.308451:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203170.308467:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203170.308472:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203170.308484:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4625 00000100:00080000:2.0:1788203170.308497:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203170.308505:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203170.308548:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187797888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203170.308576:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203170.308580:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203170.308582:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203170.308588:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4625 00000100:00080000:2.0:1788203170.308592:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203170.308596:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203170.308609:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203170.308626:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187798016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203170.308627:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187797888:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203170.308638:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203170.308642:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203170.308644:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203170.308648:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4625 00000100:00080000:2.0:1788203170.308652:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203170.308655:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203170.308660:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203170.308669:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187798144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203170.308678:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203170.308682:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203170.308684:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203170.308688:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4625 00000100:00080000:2.0:1788203170.308691:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203170.308695:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203170.308700:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203170.308710:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187798272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203170.308719:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203170.308723:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203170.308725:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203170.308728:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4625 00000100:00080000:2.0:1788203170.308731:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203170.308734:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203170.308739:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203170.308749:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187798400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203170.308784:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187798016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203170.308827:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187798144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203170.308842:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187798272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203170.308880:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187798400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203171.012541:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203172.036532:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203173.060460:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788203173.380721:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:1.0:1788203173.380770:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788203173.380789:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788203173.380856:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788203173.380872:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788203173.380883:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788203173.380892:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:2.0:1788203173.380965:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203170/real 1788203173] req@ffff8e82894e5c00 x1875062187798016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203173.380990:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187798016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203173.381009:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203173.381014:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203173.381018:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187798016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203173.381025:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203170/real 1788203173] req@ffff8e82894e4380 x1875062187798144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203173.381035:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187798144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203173.381043:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203173.381046:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203173.381049:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187798144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203173.381056:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203170/real 1788203173] req@ffff8e82894e5f80 x1875062187798272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203173.381066:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187798272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203173.381074:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203173.381077:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203173.381080:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187798272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203173.381086:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203170/real 1788203173] req@ffff8e82894e6d80 x1875062187798400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203173.381097:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187798400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203173.381104:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203173.381107:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203173.381110:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187798400:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203173.381116:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203170/real 1788203173] req@ffff8e82894e7100 x1875062187797888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203173.381127:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187797888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203173.381136:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203173.381140:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203173.381143:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187797888:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788203174.084456:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203175.108529:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203175.428422:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203175.428430:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203175.428434:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203175.428442:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4630 00000100:00080000:2.0:1788203175.428450:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203175.428456:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203175.428492:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187798528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203175.428519:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203175.428522:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203175.428529:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203175.428535:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4630 00000100:00080000:2.0:1788203175.428539:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203175.428544:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203175.428553:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203175.428569:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187798656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203175.428579:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788203175.428585:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187798528:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203175.428586:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203175.428589:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203175.428598:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4630 00000100:00080000:2.0:1788203175.428602:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203175.428607:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203175.428612:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203175.428625:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187798784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203175.428651:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203175.428655:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203175.428656:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788203175.428662:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203175.428664:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4630 00000100:00080000:2.0:1788203175.428671:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203175.428676:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203175.428683:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203175.428697:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187798912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203175.428712:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203175.428715:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203175.428720:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203175.428724:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4630 00000100:00080000:2.0:1788203175.428727:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788203175.428731:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203175/real 1788203175] req@ffff8e82894e7100 x1875062187798528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203175.428735:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203175.428744:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203175.428761:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187799040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203175.428802:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187798528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203175.428815:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203175.428821:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203175.428825:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187798528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203175.428850:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187798656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203175.428876:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203175.428891:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203175/real 1788203175] req@ffff8e82894e6d80 x1875062187798656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203175.428913:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187798656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203175.428922:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203175.428926:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203175.428930:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187798656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203175.428937:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187798784:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203175.428964:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203175.428977:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203175/real 1788203175] req@ffff8e82894e5f80 x1875062187798784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203175.428989:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187798784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203175.428997:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203175.429001:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203175.429004:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187798784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203175.429010:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187798912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203175.429030:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203175.429042:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203175/real 1788203175] req@ffff8e82894e4380 x1875062187798912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203175.429069:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187798912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203175.429077:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203175.429081:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203175.429084:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187798912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203175.429090:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187799040:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203175.429110:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203175.429121:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203175/real 1788203175] req@ffff8e82894e5c00 x1875062187799040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203175.429133:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187799040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203175.429141:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203175.429144:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203175.429147:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187799040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203176.132436:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203177.156466:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203178.180493:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203179.204499:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203180.228472:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203180.548406:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203180.548413:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203180.548417:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203180.548426:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4635 00000100:00080000:2.0:1788203180.548434:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203180.548440:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203180.548470:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187799168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203180.548494:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203180.548496:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203180.548498:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203180.548510:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4635 00000100:00080000:2.0:1788203180.548513:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203180.548516:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203180.548524:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203180.548534:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187799296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203180.548536:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187799168:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203180.548552:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203180.548554:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203180.548556:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203180.548558:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4635 00000100:00080000:2.0:1788203180.548560:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203180.548562:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203180.548565:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203180.548572:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187799424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203180.548579:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203180.548581:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203180.548583:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203180.548585:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4635 00000100:00080000:2.0:1788203180.548593:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203180.548597:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203180.548601:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788203180.548611:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203180.548618:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187799552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203180.548628:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203180.548632:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203180.548634:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203180.548640:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4635 00000100:00080000:2.0:1788203180.548647:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203180.548658:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203180.548664:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203180.548664:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203180/real 1788203180] req@ffff8e82894e5c00 x1875062187799168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203180.548700:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187799168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203180.548701:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187799680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203180.548734:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203180.548741:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203180.548750:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187799168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203180.548786:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187799296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203180.548821:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203180.548909:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203180/real 1788203180] req@ffff8e82894e4380 x1875062187799296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203180.548934:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187799296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203180.548943:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203180.548947:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203180.548951:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187799296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203180.548958:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187799424:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203180.549105:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203180.549171:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203180/real 1788203180] req@ffff8e82894e5f80 x1875062187799424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203180.549192:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187799424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203180.549201:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203180.549205:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203180.549209:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187799424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203180.549217:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187799552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203180.549236:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203180.549247:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203180/real 1788203180] req@ffff8e82894e6d80 x1875062187799552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203180.549258:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187799552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203180.549265:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203180.549268:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203180.549271:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187799552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203180.549277:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187799680:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203180.549294:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203180.549304:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203180/real 1788203180] req@ffff8e82894e7100 x1875062187799680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203180.549343:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187799680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203180.549353:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203180.549358:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203180.549366:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187799680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203181.252444:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203182.276469:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203183.300544:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203184.324526:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203185.348478:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203185.668401:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203185.668408:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203185.668412:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203185.668420:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4640 00000100:00080000:0.0:1788203185.668427:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203185.668432:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203185.668459:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187799808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203185.668478:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203185.668480:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203185.668481:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203185.668484:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4640 00000100:00080000:0.0:1788203185.668486:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203185.668488:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203185.668496:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203185.668503:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187799936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203185.668510:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203185.668512:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203185.668513:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203185.668515:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4640 00000100:00080000:0.0:1788203185.668517:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203185.668519:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203185.668522:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203185.668524:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187799808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203185.668530:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187800064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203185.668536:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203185.668538:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203185.668539:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203185.668541:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4640 00000100:00080000:0.0:1788203185.668543:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203185.668545:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203185.668547:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203185.668553:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187800192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203185.668562:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203185.668564:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203185.668565:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203185.668567:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4640 00000100:00080000:0.0:1788203185.668569:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203185.668571:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203185.668573:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203185.668581:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187800320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203185.668601:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203185.668654:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203185/real 1788203185] req@ffff8e828793d500 x1875062187799808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203185.668675:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187799808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203185.668686:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203185.668691:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203185.668695:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187799808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203185.668706:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187799936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203185.668726:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203185.668737:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203185/real 1788203185] req@ffff8e828793fb80 x1875062187799936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203185.668748:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187799936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203185.668756:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203185.668759:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203185.668762:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187799936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203185.668783:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187800064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203185.668801:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203185.668810:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203185/real 1788203185] req@ffff8e828793d880 x1875062187800064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203185.668830:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187800064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203185.668838:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203185.668841:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203185.668844:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187800064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203185.668850:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187800192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203185.668867:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203185.668876:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203185/real 1788203185] req@ffff8e828793c380 x1875062187800192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203185.668887:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187800192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203185.668894:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203185.668897:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203185.668901:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187800192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203185.668906:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187800320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203185.668921:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203185.668929:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203185/real 1788203185] req@ffff8e828793e300 x1875062187800320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203185.668940:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187800320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203185.668947:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203185.668950:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203185.668953:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187800320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203186.372456:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203187.396529:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203188.420497:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203189.444549:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203190.468541:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203190.788538:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203190.788547:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203190.788551:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203190.788560:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4645 00000100:00080000:3.0:1788203190.788570:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203190.788576:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203190.788614:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187800448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203190.788645:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203190.788648:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203190.788650:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203190.788654:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4645 00000100:00080000:3.0:1788203190.788657:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203190.788661:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203190.788670:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203190.788682:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187800576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203190.788701:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187800448:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788203190.788705:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203190.788716:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203190.788718:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203190.788722:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4645 00000100:00080000:3.0:1788203190.788725:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203190.788729:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203190.788733:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203190.788746:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187800704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203190.788754:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203190.788757:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203190.788758:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203190.788761:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4645 00000100:00080000:3.0:1788203190.788764:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203190.788769:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203190.788775:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203190.788787:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187800832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203190.788796:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788203190.788804:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788203190.788805:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203190.788813:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203190.788819:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4645 00000100:00080000:3.0:1788203190.788824:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203190.788828:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203190.788834:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203190.788849:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187800960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203190.788873:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203190/real 1788203190] req@ffff8e828953b100 x1875062187800448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203190.788897:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187800448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203190.788910:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203190.788914:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203190.788919:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187800448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203190.788947:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187800576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203190.788973:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203190.788987:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203190/real 1788203190] req@ffff8e828953a680 x1875062187800576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203190.789000:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187800576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203190.789009:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203190.789012:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203190.789015:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187800576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203190.789022:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187800704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203190.789042:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203190.789053:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203190/real 1788203190] req@ffff8e8289538a80 x1875062187800704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203190.789065:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187800704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203190.789073:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203190.789076:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203190.789080:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187800704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203190.789086:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187800832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203190.789107:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203190.789118:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203190/real 1788203190] req@ffff8e8289539500 x1875062187800832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203190.789137:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187800832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203190.789145:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203190.789148:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203190.789151:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187800832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203190.789157:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187800960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203190.789175:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203190.789186:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203190/real 1788203190] req@ffff8e8289539880 x1875062187800960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203190.789197:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187800960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203190.789204:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203190.789208:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203190.789211:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187800960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203191.492503:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203192.516524:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203193.540503:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203194.564451:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203195.588502:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203195.908431:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203195.908441:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203195.908446:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203195.908456:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4650 00000100:00080000:3.0:1788203195.908467:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203195.908474:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203195.908514:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187801088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203195.908539:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203195.908542:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203195.908544:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203195.908547:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4650 00000100:00080000:3.0:1788203195.908549:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203195.908551:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203195.908560:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203195.908574:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187801216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203195.908585:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203195.908591:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203195.908593:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203195.908597:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4650 00000100:00080000:3.0:1788203195.908601:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788203195.908605:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187801088:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788203195.908608:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203195.908616:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203195.908626:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187801344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203195.908633:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203195.908636:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203195.908638:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203195.908642:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4650 00000100:00080000:3.0:1788203195.908645:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203195.908649:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203195.908654:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203195.908666:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187801472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203195.908676:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203195.908680:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203195.908682:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203195.908685:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4650 00000100:00080000:3.0:1788203195.908688:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203195.908693:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:0.0:1788203195.908696:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:3.0:1788203195.908699:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203195.908709:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187801600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203195.908766:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203195/real 1788203195] req@ffff8e8289539f80 x1875062187801088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203195.908798:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187801088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203195.908808:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203195.908812:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203195.908816:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187801088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203195.908824:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187801216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203195.908843:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203195.908854:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203195/real 1788203195] req@ffff8e8289538380 x1875062187801216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203195.908864:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187801216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203195.908871:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203195.908874:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203195.908877:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187801216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203195.908902:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187801344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203195.908919:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203195.908928:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203195/real 1788203195] req@ffff8e828953bb80 x1875062187801344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203195.908949:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187801344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203195.908961:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203195.908963:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203195.908966:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187801344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203195.908971:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187801472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203195.908987:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203195.908996:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203195/real 1788203195] req@ffff8e8289539880 x1875062187801472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203195.909004:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187801472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203195.909011:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203195.909013:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203195.909016:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187801472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203195.909020:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187801600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203195.909058:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203195.909099:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203195/real 1788203195] req@ffff8e8289539500 x1875062187801600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203195.909109:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187801600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203195.909115:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203195.909118:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203195.909121:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187801600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203196.612511:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203197.636471:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203198.660557:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203199.684546:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203200.708486:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203201.028441:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203201.028451:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203201.028456:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203201.028464:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4656 00000100:00080000:3.0:1788203201.028473:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203201.028479:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203201.028508:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187801728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203201.028530:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203201.028533:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203201.028534:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203201.028537:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4656 00000100:00080000:3.0:1788203201.028539:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203201.028542:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203201.028550:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203201.028564:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187801856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203201.028569:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203201.028572:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203201.028573:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203201.028575:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4656 00000100:00080000:3.0:1788203201.028578:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203201.028581:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203201.028587:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203201.028598:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187801728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203201.028599:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187801984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203201.028622:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203201.028624:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203201.028626:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203201.028637:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4656 00000100:00080000:3.0:1788203201.028642:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203201.028646:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203201.028651:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203201.028667:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187802112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203201.028676:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203201.028679:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203201.028680:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203201.028684:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4656 00000100:00080000:3.0:1788203201.028688:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203201.028692:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203201.028698:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788203201.028703:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203201.028718:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187802240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203201.028769:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203201/real 1788203201] req@ffff8e8289538a80 x1875062187801728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203201.028793:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187801728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203201.028805:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203201.028810:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203201.028815:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187801728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203201.028824:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187801856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203201.028847:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203201.028859:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203201/real 1788203201] req@ffff8e828953a680 x1875062187801856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203201.028871:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187801856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203201.028880:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203201.028883:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203201.028887:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187801856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203201.028916:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187801984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203201.028936:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203201.028947:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203201/real 1788203201] req@ffff8e828953b100 x1875062187801984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203201.028959:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187801984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203201.028967:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203201.028970:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203201.028974:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187801984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203201.028981:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187802112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203201.028999:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203201.029009:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203201/real 1788203201] req@ffff8e8289539500 x1875062187802112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203201.029019:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187802112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203201.029027:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203201.029030:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203201.029033:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187802112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203201.029039:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187802240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203201.029058:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203201.029068:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203201/real 1788203201] req@ffff8e8289539880 x1875062187802240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203226 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203201.029080:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187802240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203226 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203201.029088:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203201.029091:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203201.029094:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187802240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203201.732541:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203202.756496:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203203.780499:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203204.804468:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203205.828430:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203206.148440:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203206.148446:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203206.148449:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203206.148456:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4661 00000100:00080000:2.0:1788203206.148466:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203206.148471:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203206.148499:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187802368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203206.148518:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203206.148520:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203206.148522:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203206.148524:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4661 00000100:00080000:2.0:1788203206.148526:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203206.148528:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203206.148534:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203206.148543:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187802496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203206.148558:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203206.148561:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203206.148564:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203206.148567:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4661 00000100:00080000:2.0:1788203206.148571:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203206.148574:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203206.148579:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203206.148588:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187802624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203206.148596:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203206.148598:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203206.148600:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203206.148605:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4661 00000100:00100000:0.0:1788203206.148606:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187802368:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203206.148608:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203206.148614:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203206.148624:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203206.148631:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187802752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203206.148637:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203206.148639:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203206.148640:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203206.148642:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4661 00000100:00080000:2.0:1788203206.148645:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203206.148647:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203206.148650:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203206.148655:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187802880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203206.148706:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203206.148763:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203206/real 1788203206] req@ffff8e82894e6300 x1875062187802368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203206.148778:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187802368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203206.148790:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203206.148796:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203206.148801:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187802368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203206.148808:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187802496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203206.148826:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203206.148836:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203206/real 1788203206] req@ffff8e82894e7800 x1875062187802496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203206.148853:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187802496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203206.148862:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203206.148865:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203206.148868:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187802496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203206.148898:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187802624:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203206.148918:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203206.148928:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203206/real 1788203206] req@ffff8e82894e4a80 x1875062187802624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203206.148940:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187802624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203206.148948:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203206.148951:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203206.148955:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187802624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203206.148961:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187802752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203206.148982:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203206.148993:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203206/real 1788203206] req@ffff8e82894e4e00 x1875062187802752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203206.149003:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187802752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203206.149017:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203206.149021:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203206.149024:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187802752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203206.149031:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187802880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203206.149051:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203206.149061:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203206/real 1788203206] req@ffff8e82894e4000 x1875062187802880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203231 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203206.149079:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187802880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203231 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203206.149088:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203206.149091:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203206.149094:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187802880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203206.852476:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203207.876435:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203208.900453:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203209.924452:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203210.948465:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203211.268420:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203211.268428:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203211.268431:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203211.268439:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4666 00000100:00080000:0.0:1788203211.268447:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203211.268452:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203211.268481:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187803008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203211.268510:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203211.268518:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203211.268519:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203211.268523:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4666 00000100:00080000:0.0:1788203211.268527:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203211.268531:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203211.268539:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203211.268551:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187803136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203211.268559:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203211.268562:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788203211.268563:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187803008:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203211.268564:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203211.268569:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4666 00000100:00080000:0.0:1788203211.268575:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203211.268577:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203211.268580:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203211.268588:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187803264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203211.268598:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203211.268600:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203211.268601:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203211.268604:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4666 00000100:00080000:0.0:1788203211.268606:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203211.268608:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203211.268612:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203211.268617:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187803392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203211.268623:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203211.268624:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203211.268626:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203211.268627:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4666 00000100:00080000:0.0:1788203211.268629:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203211.268634:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203211.268636:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203211.268644:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187803520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203211.268644:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203211.268697:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203211/real 1788203211] req@ffff8e828793f480 x1875062187803008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203211.268717:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187803008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203211.268727:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203211.268731:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203211.268734:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187803008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203211.268756:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187803136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203211.268779:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203211.268792:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203211/real 1788203211] req@ffff8e828793dc00 x1875062187803136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203211.268804:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187803136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203211.268811:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203211.268814:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203211.268817:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187803136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203211.268822:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187803264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203211.268842:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203211.268853:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203211/real 1788203211] req@ffff8e828793c000 x1875062187803264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203211.268865:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187803264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203211.268873:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203211.268875:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203211.268879:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187803264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203211.268884:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187803392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203211.268905:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203211.268916:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203211/real 1788203211] req@ffff8e828793ca80 x1875062187803392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203211.268927:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187803392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203211.268934:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203211.268938:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203211.268942:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187803392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203211.268948:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187803520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203211.268966:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203211.268976:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203211/real 1788203211] req@ffff8e828793c700 x1875062187803520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203236 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203211.268986:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187803520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203236 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203211.268994:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203211.268996:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203211.268999:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187803520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203211.972484:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203212.996549:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203214.020432:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203215.044483:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203216.068440:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203216.388401:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203216.388409:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203216.388413:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203216.388420:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4671 00000100:00080000:0.0:1788203216.388428:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203216.388433:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203216.388459:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187803648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203216.388480:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203216.388482:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203216.388485:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203216.388487:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4671 00000100:00080000:0.0:1788203216.388490:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203216.388493:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203216.388501:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203216.388514:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187803648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203216.388521:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187803776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203216.388529:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203216.388531:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203216.388532:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203216.388535:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4671 00000100:00080000:0.0:1788203216.388538:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203216.388540:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203216.388544:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203216.388553:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187803904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203216.388559:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203216.388562:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203216.388564:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203216.388568:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4671 00000100:00080000:0.0:1788203216.388572:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788203216.388576:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203216.388579:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203216.388586:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203216.388599:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187804032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203216.388608:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788203216.388610:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203216/real 1788203216] req@ffff8e828793ce00 x1875062187803648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203216.388615:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203216.388620:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203216.388626:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4671 00000100:00100000:2.0:1788203216.388629:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187803648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203216.388632:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203216.388641:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:2.0:1788203216.388644:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788203216.388647:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788203216.388651:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203216.388657:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187803648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203216.388663:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187804160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203216.388680:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187803776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203216.388700:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203216.388709:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203216/real 1788203216] req@ffff8e828793f800 x1875062187803776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203216.388717:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187803776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203216.388722:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203216.388724:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203216.388727:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187803776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203216.388731:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187803904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203216.388740:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203216.388745:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203216/real 1788203216] req@ffff8e828793e680 x1875062187803904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203216.388752:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187803904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203216.388756:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203216.388757:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203216.388759:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187803904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203216.388762:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187804032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203216.388771:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203216.388776:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203216/real 1788203216] req@ffff8e828793f100 x1875062187804032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203216.388782:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187804032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203216.388786:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203216.388787:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203216.388789:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187804032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203216.388792:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187804160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203216.388799:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203216.388804:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203216/real 1788203216] req@ffff8e828793c700 x1875062187804160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203216.388810:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187804160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203216.388815:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203216.388817:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203216.388818:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187804160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203217.092483:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203218.116490:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203219.140523:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203220.164418:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203221.188502:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203221.508441:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203221.508448:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203221.508452:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203221.508460:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4676 00000100:00080000:2.0:1788203221.508468:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203221.508474:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203221.508502:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187804288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203221.508523:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203221.508525:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203221.508526:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203221.508529:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4676 00000100:00080000:2.0:1788203221.508533:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203221.508535:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203221.508542:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203221.508551:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187804416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203221.508558:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203221.508560:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203221.508562:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203221.508564:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4676 00000100:00080000:2.0:1788203221.508565:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203221.508569:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203221.508577:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187804288:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788203221.508578:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203221.508594:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187804544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203221.508601:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203221.508604:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203221.508605:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203221.508607:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4676 00000100:00080000:2.0:1788203221.508609:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203221.508612:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203221.508615:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203221.508621:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187804672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203221.508633:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203221.508637:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203221.508638:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203221.508640:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4676 00000100:00080000:2.0:1788203221.508649:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203221.508653:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203221.508656:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788203221.508666:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203221.508672:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187804800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203221.508725:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203221/real 1788203221] req@ffff8e82894e6a00 x1875062187804288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203221.508744:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187804288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203221.508754:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203221.508759:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203221.508763:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187804288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203221.508792:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187804416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203221.508815:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203221.508829:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203221/real 1788203221] req@ffff8e82894e5880 x1875062187804416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203221.508842:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187804416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203221.508851:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203221.508855:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203221.508858:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187804416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203221.508864:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187804544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203221.508885:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203221.508896:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203221/real 1788203221] req@ffff8e82894e6680 x1875062187804544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203221.508908:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187804544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203221.508915:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203221.508919:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203221.508922:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187804544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203221.508929:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187804672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203221.508949:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203221.508960:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203221/real 1788203221] req@ffff8e82894e7480 x1875062187804672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203221.509002:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187804672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203221.509010:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203221.509013:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203221.509016:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187804672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203221.509022:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187804800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203221.509041:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203221.509052:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203221/real 1788203221] req@ffff8e82894e4000 x1875062187804800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203221.509063:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187804800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203221.509068:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203221.509070:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203221.509072:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187804800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203222.212457:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203223.236459:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203224.260420:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203225.284445:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203226.308461:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203226.628382:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203226.628389:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203226.628392:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203226.628399:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4681 00000100:00080000:2.0:1788203226.628407:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203226.628412:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203226.628448:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187804928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203226.628467:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203226.628470:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203226.628471:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203226.628473:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4681 00000100:00080000:2.0:1788203226.628475:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203226.628477:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203226.628484:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203226.628493:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187805056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203226.628499:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203226.628503:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203226.628504:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203226.628506:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4681 00000100:00080000:2.0:1788203226.628508:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203226.628511:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203226.628514:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203226.628524:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187805184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203226.628532:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187804928:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203226.628535:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203226.628539:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203226.628541:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203226.628545:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4681 00000100:00080000:2.0:1788203226.628550:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203226.628553:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203226.628559:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203226.628569:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187805312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203226.628575:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203226.628578:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203226.628579:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203226.628581:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4681 00000100:00080000:2.0:1788203226.628584:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203226.628589:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:3.0:1788203226.628589:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:2.0:1788203226.628595:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203226.628603:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187805440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203226.628629:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203226/real 1788203226] req@ffff8e82894e4e00 x1875062187804928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203226.628646:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187804928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203226.628664:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203226.628670:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203226.628673:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187804928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203226.628678:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187805056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203226.628695:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203226.628703:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203226/real 1788203226] req@ffff8e82894e4a80 x1875062187805056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203226.628710:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187805056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203226.628719:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203226.628721:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203226.628723:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187805056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203226.628739:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187805184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203226.628749:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203226.628755:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203226/real 1788203226] req@ffff8e82894e7800 x1875062187805184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203226.628761:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187805184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203226.628766:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203226.628767:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203226.628769:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187805184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203226.628772:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187805312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203226.628781:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203226.628786:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203226/real 1788203226] req@ffff8e82894e6300 x1875062187805312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203226.628792:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187805312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203226.628796:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203226.628798:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203226.628800:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187805312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203226.628803:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187805440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203226.628812:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203226.628819:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203226/real 1788203226] req@ffff8e82894e7100 x1875062187805440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203226.628829:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187805440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203226.628837:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203226.628839:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203226.628843:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187805440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203227.332450:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203228.356467:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203229.380450:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203230.404453:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203231.428447:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203231.748413:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203231.748423:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203231.748428:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203231.748437:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4686 00000100:00080000:0.0:1788203231.748447:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203231.748456:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203231.748502:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187805568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203231.748530:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203231.748538:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203231.748539:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203231.748543:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4686 00000100:00080000:0.0:1788203231.748546:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203231.748550:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203231.748559:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203231.748572:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187805696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203231.748572:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187805568:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203231.748587:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203231.748590:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203231.748592:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203231.748596:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4686 00000100:00080000:0.0:1788203231.748600:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203231.748603:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203231.748608:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203231.748618:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187805824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203231.748627:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203231.748630:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203231.748633:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788203231.748633:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203231.748639:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4686 00000100:00080000:0.0:1788203231.748644:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203231.748647:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203231.748651:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203231.748666:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187805952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203231.748673:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203231.748677:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788203231.748677:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203231/real 1788203231] req@ffff8e828793ca80 x1875062187805568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203231.748680:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203231.748687:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4686 00000100:00080000:0.0:1788203231.748692:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203231.748696:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203231.748699:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203231.748701:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187805568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203231.748708:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187806080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203231.748709:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203231.748717:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203231.748723:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187805568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203231.748740:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187805696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203231.748753:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203231.748762:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203231/real 1788203231] req@ffff8e828793c000 x1875062187805696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203231.748769:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187805696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203231.748774:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203231.748775:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203231.748777:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187805696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203231.748780:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187805824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203231.748814:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203231.748822:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203231/real 1788203231] req@ffff8e828793dc00 x1875062187805824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203231.748828:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187805824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203231.748833:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203231.748835:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203231.748837:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187805824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203231.748840:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187805952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203231.748851:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203231.748856:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203231/real 1788203231] req@ffff8e828793f480 x1875062187805952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203231.748862:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187805952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203231.748867:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203231.748869:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203231.748871:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187805952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203231.748873:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187806080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203231.748882:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203231.748886:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203231/real 1788203231] req@ffff8e828793e300 x1875062187806080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203231.748892:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187806080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203231.748897:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203231.748898:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203231.748900:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187806080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203232.452464:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203233.476490:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203234.500483:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203235.524461:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203236.548457:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203236.868400:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203236.868411:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203236.868416:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203236.868425:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4691 00000100:00080000:0.0:1788203236.868436:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203236.868442:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203236.868482:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187806208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203236.868505:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203236.868508:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203236.868509:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203236.868513:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4691 00000100:00080000:0.0:1788203236.868516:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203236.868519:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203236.868528:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203236.868537:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187806336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203236.868543:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187806208:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203236.868545:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203236.868550:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203236.868551:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203236.868554:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4691 00000100:00080000:0.0:1788203236.868556:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203236.868558:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203236.868560:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203236.868568:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187806464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203236.868573:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203236.868575:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203236.868576:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203236.868578:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4691 00000100:00080000:0.0:1788203236.868580:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203236.868583:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203236.868587:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203236.868592:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187806592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203236.868599:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203236.868602:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203236.868603:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203236.868605:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4691 00000100:00080000:0.0:1788203236.868607:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203236.868609:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203236.868612:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203236.868619:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187806720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203236.868666:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187806336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203236.868713:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187806464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203236.868736:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187806592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203236.868750:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187806720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203237.572456:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203238.596481:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203239.620434:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788203239.940550:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:2.0:1788203239.940583:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788203239.940598:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788203239.940633:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788203239.940643:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788203239.940664:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788203239.940674:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:3.0:1788203239.940694:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203236/real 1788203239] req@ffff8e828793d880 x1875062187806336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203239.940725:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187806336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203239.940736:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203239.940741:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203239.940745:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187806336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203239.940753:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203236/real 1788203239] req@ffff8e828793fb80 x1875062187806464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203239.940765:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187806464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203239.940772:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203239.940776:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203239.940779:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187806464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203239.940785:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203236/real 1788203239] req@ffff8e828793d500 x1875062187806592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203239.940797:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187806592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203239.940805:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203239.940808:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203239.940811:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187806592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203239.940816:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203236/real 1788203239] req@ffff8e828793e300 x1875062187806720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203239.940827:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187806720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203239.940835:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203239.940838:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203239.940841:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187806720:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203239.940847:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203236/real 1788203239] req@ffff8e828793c380 x1875062187806208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203239.940859:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187806208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203239.940866:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203239.940870:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203239.940872:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187806208:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788203240.644449:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203241.668494:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203241.988394:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203241.988403:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203241.988405:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203241.988413:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4697 00000100:00080000:0.0:1788203241.988421:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203241.988427:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203241.988453:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187806848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203241.988474:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203241.988476:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203241.988477:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203241.988480:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4697 00000100:00080000:0.0:1788203241.988482:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203241.988484:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203241.988491:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203241.988499:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187806976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203241.988509:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203241.988512:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203241.988514:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203241.988518:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4697 00000100:00080000:0.0:1788203241.988521:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203241.988527:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203241.988531:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187806848:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788203241.988533:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203241.988545:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187807104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203241.988556:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203241.988559:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203241.988561:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203241.988564:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4697 00000100:00080000:0.0:1788203241.988568:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203241.988573:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203241.988578:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203241.988587:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187807232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203241.988602:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788203241.988602:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203241.988619:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203241.988622:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203241.988627:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4697 00000100:00080000:0.0:1788203241.988632:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203241.988637:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203241.988642:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203241.988654:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187807360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203241.988677:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203241/real 1788203241] req@ffff8e828793f480 x1875062187806848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203241.988695:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187806848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203241.988704:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203241.988708:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203241.988712:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187806848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203241.988719:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187806976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203241.988742:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203241.988754:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203241/real 1788203241] req@ffff8e828793dc00 x1875062187806976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203241.988763:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187806976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203241.988769:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203241.988773:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203241.988775:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187806976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203241.988796:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187807104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203241.988817:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203241.988827:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203241/real 1788203241] req@ffff8e828793c000 x1875062187807104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203241.988837:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187807104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203241.988850:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203241.988853:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203241.988856:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187807104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203241.988862:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187807232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203241.988885:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203241.988896:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203241/real 1788203241] req@ffff8e828793ca80 x1875062187807232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203241.988906:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187807232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203241.988913:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203241.988916:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203241.988918:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187807232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203241.988923:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187807360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203241.988939:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203241.988948:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203241/real 1788203241] req@ffff8e828793c700 x1875062187807360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203241.988957:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187807360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203241.988963:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203241.988966:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203241.988968:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187807360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203242.692487:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203243.716451:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203244.740458:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203245.764454:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203246.788429:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203247.108406:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203247.108413:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203247.108416:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203247.108423:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4702 00000100:00080000:1.0:1788203247.108431:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203247.108435:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203247.108461:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187807488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203247.108478:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203247.108484:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203247.108485:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203247.108488:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4702 00000100:00080000:1.0:1788203247.108490:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203247.108494:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203247.108501:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203247.108511:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187807616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203247.108511:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187807488:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788203247.108537:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203247.108540:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203247.108541:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203247.108544:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4702 00000100:00080000:1.0:1788203247.108546:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203247.108550:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203247.108555:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203247.108567:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187807744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203247.108576:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788203247.108577:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788203247.108583:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203247.108585:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203247.108590:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4702 00000100:00080000:1.0:1788203247.108595:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203247.108598:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203247.108603:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203247.108610:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203247/real 1788203247] req@ffff8e828d720e00 x1875062187807488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203247.108618:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187807872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203247.108626:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187807488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203247.108631:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203247.108639:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203247.108642:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203247.108650:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4702 00000100:00080000:3.0:1788203247.108653:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203247.108655:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203247.108660:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:1.0:1788203247.108663:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203247.108669:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187807488:192.168.201.121@tcp:250:kworker.0 00000008:00100000:1.0:1788203247.108672:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203247.108686:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187808000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203247.108693:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187807616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203247.108721:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203247.108730:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203247/real 1788203247] req@ffff8e828d723800 x1875062187807616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203247.108737:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187807616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203247.108742:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203247.108744:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203247.108746:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187807616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203247.108749:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187807744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203247.108759:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203247.108767:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203247/real 1788203247] req@ffff8e828d721500 x1875062187807744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203247.108778:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187807744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203247.108784:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203247.108787:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203247.108789:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187807744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203247.108794:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187807872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203247.108812:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203247.108821:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203247/real 1788203247] req@ffff8e828d720000 x1875062187807872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203247.108830:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187807872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203247.108837:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203247.108840:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203247.108843:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187807872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203247.108857:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187808000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203247.108874:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203247.108881:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203247/real 1788203247] req@ffff8e828d722680 x1875062187808000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203247.108887:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187808000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203272 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203247.108892:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203247.108893:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203247.108895:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187808000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203247.812547:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203248.836441:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203249.860445:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203250.884437:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203251.908490:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203252.228403:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203252.228411:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203252.228415:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203252.228423:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4707 00000100:00080000:1.0:1788203252.228442:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203252.228449:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203252.228476:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187808128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203252.228496:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203252.228499:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203252.228501:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203252.228505:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4707 00000100:00080000:1.0:1788203252.228509:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203252.228512:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203252.228522:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203252.228536:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187808256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203252.228545:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203252.228549:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203252.228551:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203252.228555:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4707 00000100:00080000:1.0:1788203252.228559:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203252.228564:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203252.228569:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203252.228583:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187808384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203252.228592:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187808128:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788203252.228597:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203252.228601:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203252.228603:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203252.228609:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4707 00000100:00080000:1.0:1788203252.228612:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203252.228616:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203252.228621:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203252.228629:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187808512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203252.228635:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203252.228637:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203252.228638:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203252.228640:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4707 00000100:00080000:1.0:1788203252.228643:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203252.228644:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:0.0:1788203252.228650:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:1.0:1788203252.228650:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203252.228659:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187808640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203252.228697:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203252/real 1788203252] req@ffff8e828d722680 x1875062187808128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203252.228712:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187808128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203252.228720:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203252.228723:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203252.228726:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187808128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203252.228732:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187808256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203252.228749:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203252.228757:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203252/real 1788203252] req@ffff8e828d720000 x1875062187808256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203252.228768:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187808256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203252.228773:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203252.228775:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203252.228777:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187808256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203252.228792:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187808384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203252.228802:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203252.228808:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203252/real 1788203252] req@ffff8e828d721500 x1875062187808384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203252.228814:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187808384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203252.228819:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203252.228821:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203252.228822:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187808384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203252.228826:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187808512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203252.228836:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203252.228841:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203252/real 1788203252] req@ffff8e828d723800 x1875062187808512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203252.228847:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187808512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203252.228856:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203252.228859:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203252.228862:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187808512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203252.228868:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187808640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203252.228885:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203252.228896:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203252/real 1788203252] req@ffff8e828d720e00 x1875062187808640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203277 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203252.228907:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187808640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203277 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203252.228916:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203252.228920:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203252.228924:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187808640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203252.932437:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203253.956529:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203254.980497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203256.004473:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203257.028440:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203257.348424:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203257.348433:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203257.348438:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203257.348447:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4712 00000100:00080000:1.0:1788203257.348457:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203257.348464:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203257.348492:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187808768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203257.348514:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203257.348518:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203257.348520:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203257.348524:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4712 00000100:00080000:1.0:1788203257.348528:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203257.348532:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203257.348541:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203257.348554:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187808768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203257.348557:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187808896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203257.348570:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203257.348574:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203257.348577:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203257.348580:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4712 00000100:00080000:1.0:1788203257.348596:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203257.348601:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:0.0:1788203257.348605:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:1.0:1788203257.348608:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203257.348621:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187809024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203257.348628:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203257.348630:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203257.348631:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203257.348634:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4712 00000100:00080000:1.0:1788203257.348637:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788203257.348640:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203257/real 1788203257] req@ffff8e828d720380 x1875062187808768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203257.348642:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203257.348647:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203257.348657:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187809152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203257.348663:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788203257.348666:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187808768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203257.348666:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203257.348670:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203257.348675:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4712 00000100:00080000:1.0:1788203257.348680:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203257.348685:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:0.0:1788203257.348688:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1788203257.348689:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788203257.348695:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203257.348699:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187809280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203257.348700:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187808768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203257.348724:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187808896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203257.348740:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203257.348748:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203257/real 1788203257] req@ffff8e828d722300 x1875062187808896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203257.348755:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187808896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203257.348760:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203257.348762:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203257.348764:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187808896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203257.348767:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187809024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203257.348776:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203257.348781:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203257/real 1788203257] req@ffff8e828d722d80 x1875062187809024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203257.348787:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187809024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203257.348791:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203257.348793:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203257.348794:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187809024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203257.348797:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187809152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203257.348806:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203257.348811:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203257/real 1788203257] req@ffff8e828d721180 x1875062187809152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203257.348817:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187809152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203257.348822:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203257.348823:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203257.348825:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187809152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203257.348827:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187809280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203257.348835:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203257.348840:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203257/real 1788203257] req@ffff8e828d720a80 x1875062187809280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203257.348845:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187809280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203257.348850:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203257.348851:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203257.348852:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187809280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203258.052470:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203259.076448:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203260.100516:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203261.124525:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203262.148492:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203262.468465:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203262.468473:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203262.468478:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203262.468494:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4717 00000100:00080000:0.0:1788203262.468504:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203262.468511:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203262.468552:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187809408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203262.468577:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203262.468580:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203262.468582:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203262.468591:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4717 00000100:00080000:0.0:1788203262.468595:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203262.468599:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203262.468606:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187809408:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788203262.468608:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203262.468624:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187809536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203262.468635:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203262.468638:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203262.468640:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203262.468643:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4717 00000100:00080000:0.0:1788203262.468648:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203262.468652:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203262.468657:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203262.468666:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203262.468670:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187809664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203262.468683:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203262.468685:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203262.468687:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203262.468690:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4717 00000100:00080000:0.0:1788203262.468692:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203262.468695:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203262.468698:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203262.468707:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187809792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203262.468711:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203262/real 1788203262] req@ffff8e828793f100 x1875062187809408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203262.468716:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203262.468723:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203262.468726:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203262.468732:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4717 00000100:00080000:0.0:1788203262.468734:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203262.468737:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203262.468739:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203262.468744:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187809408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203262.468748:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187809920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203262.468766:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203262.468772:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203262.468777:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187809408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203262.468797:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187809536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203262.468813:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203262.468822:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203262/real 1788203262] req@ffff8e828793e680 x1875062187809536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203262.468829:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187809536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203262.468834:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203262.468836:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203262.468838:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187809536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203262.468841:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187809664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203262.468850:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203262.468855:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203262/real 1788203262] req@ffff8e828793f800 x1875062187809664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203262.468869:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187809664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203262.468878:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203262.468879:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203262.468881:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187809664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203262.468884:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187809792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203262.468895:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203262.468900:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203262/real 1788203262] req@ffff8e828793ce00 x1875062187809792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203262.468906:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187809792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203262.468910:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203262.468912:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203262.468914:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187809792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203262.468917:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187809920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203262.468925:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203262.468930:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203262/real 1788203262] req@ffff8e828793c700 x1875062187809920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203262.468936:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187809920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203262.468940:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203262.468942:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203262.468944:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187809920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203263.172475:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203264.196506:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203265.220529:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203266.244525:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203267.268513:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203267.588431:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203267.588444:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203267.588450:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203267.588463:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4722 00000100:00080000:0.0:1788203267.588476:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203267.588485:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203267.588549:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187810048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203267.588583:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203267.588588:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203267.588590:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203267.588596:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4722 00000100:00080000:0.0:1788203267.588602:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203267.588608:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203267.588620:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203267.588633:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187810048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203267.588641:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187810176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203267.588657:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203267.588660:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203267.588661:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203267.588665:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4722 00000100:00080000:0.0:1788203267.588668:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203267.588671:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203267.588676:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203267.588689:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187810304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203267.588702:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203267.588705:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203267.588706:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203267.588710:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4722 00000100:00080000:0.0:1788203267.588715:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203267.588718:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203267.588723:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788203267.588727:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203267.588740:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187810432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203267.588749:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203267.588752:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203267.588754:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203267.588758:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4722 00000100:00080000:0.0:1788203267.588762:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203267.588767:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203267.588773:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203267.588781:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203267/real 1788203267] req@ffff8e828793ca80 x1875062187810048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203267.588791:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187810560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203267.588801:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187810048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203267.588831:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203267.588837:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203267.588840:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187810048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203267.588864:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187810176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203267.588881:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203267.588889:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203267/real 1788203267] req@ffff8e828793c000 x1875062187810176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203267.588897:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187810176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203267.588903:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203267.588905:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203267.588908:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187810176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203267.588914:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187810304:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203267.588932:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203267.588951:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203267/real 1788203267] req@ffff8e828793dc00 x1875062187810304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203267.588965:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187810304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203267.588973:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203267.588977:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203267.588980:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187810304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203267.588987:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187810432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203267.589008:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203267.589019:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203267/real 1788203267] req@ffff8e828793f480 x1875062187810432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203267.589031:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187810432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203267.589039:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203267.589042:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203267.589046:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187810432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203267.589052:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187810560:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203267.589069:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203267.589090:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203267/real 1788203267] req@ffff8e828793c380 x1875062187810560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203267.589102:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187810560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203267.589110:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203267.589114:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203267.589117:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187810560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203268.292462:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203269.316466:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203270.340469:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203271.364494:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203272.388438:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203272.708470:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203272.708483:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203272.708488:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203272.708499:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4727 00000100:00080000:2.0:1788203272.708511:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203272.708518:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203272.708552:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187810688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203272.708574:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203272.708576:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203272.708578:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203272.708581:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4727 00000100:00080000:2.0:1788203272.708583:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203272.708586:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203272.708594:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203272.708603:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187810816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203272.708611:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203272.708615:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203272.708618:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203272.708618:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187810688:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203272.708622:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4727 00000100:00080000:2.0:1788203272.708627:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203272.708631:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203272.708635:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203272.708642:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187810944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203272.708648:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203272.708650:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203272.708652:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203272.708655:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4727 00000100:00080000:2.0:1788203272.708657:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203272.708659:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203272.708671:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203272.708678:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187811072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203272.708688:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788203272.708694:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203272.708695:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203272.708700:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203272.708706:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4727 00000100:00080000:2.0:1788203272.708711:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203272.708717:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203272.708722:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203272.708737:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187811200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203272.708745:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203272/real 1788203272] req@ffff8e82894e6d80 x1875062187810688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203272.708767:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187810688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203272.708778:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203272.708783:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203272.708787:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187810688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203272.708811:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187810816:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203272.708836:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203272.708851:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203272/real 1788203272] req@ffff8e82894e5f80 x1875062187810816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203272.708864:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187810816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203272.708872:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203272.708876:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203272.708879:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187810816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203272.708886:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187810944:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203272.708907:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203272.708920:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203272/real 1788203272] req@ffff8e82894e4380 x1875062187810944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203272.708933:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187810944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203272.708941:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203272.708945:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203272.708948:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187810944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203272.708955:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187811072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203272.708976:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203272.708987:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203272/real 1788203272] req@ffff8e82894e5c00 x1875062187811072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203272.708999:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187811072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203272.709006:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203272.709009:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203272.709013:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187811072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203272.709018:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187811200:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203272.709037:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203272.709048:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203272/real 1788203272] req@ffff8e82894e7100 x1875062187811200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203272.709059:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187811200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203272.709067:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203272.709070:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203272.709074:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187811200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203273.412504:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203274.436464:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203275.460511:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203276.484464:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203277.508472:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203277.828443:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203277.828454:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203277.828459:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203277.828467:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4732 00000100:00080000:0.0:1788203277.828486:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203277.828492:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203277.828524:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187811328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203277.828546:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203277.828549:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203277.828551:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203277.828553:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4732 00000100:00080000:0.0:1788203277.828556:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203277.828559:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203277.828566:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203277.828582:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187811328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203277.828583:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187811456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203277.828600:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203277.828603:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203277.828605:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203277.828610:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4732 00000100:00080000:0.0:1788203277.828614:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203277.828619:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203277.828625:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203277.828638:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187811584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203277.828649:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788203277.828653:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203277.828655:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203277.828658:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203277.828662:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4732 00000100:00080000:0.0:1788203277.828666:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203277.828670:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203277.828675:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203277.828684:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187811712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203277.828691:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203277.828693:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203277.828695:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203277.828697:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4732 00000100:00100000:3.0:1788203277.828701:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203277/real 1788203277] req@ffff8e828793e300 x1875062187811328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203277.828702:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203277.828709:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203277.828715:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203277.828727:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187811328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203277.828729:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187811840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203277.828748:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203277.828753:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203277.828757:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187811328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203277.828776:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187811456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203277.828792:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203277.828800:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203277/real 1788203277] req@ffff8e828793d500 x1875062187811456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203277.828808:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187811456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203277.828817:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203277.828819:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203277.828822:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187811456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203277.828825:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187811584:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203277.828835:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203277.828841:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203277/real 1788203277] req@ffff8e828793fb80 x1875062187811584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203277.828848:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187811584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203277.828853:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203277.828855:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203277.828857:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187811584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203277.828860:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187811712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203277.828870:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203277.828875:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203277/real 1788203277] req@ffff8e828793d880 x1875062187811712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203277.828881:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187811712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203277.828886:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203277.828888:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203277.828890:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187811712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203277.828894:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187811840:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203277.828903:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203277.828908:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203277/real 1788203277] req@ffff8e828793c380 x1875062187811840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203277.828915:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187811840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203277.828920:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203277.828921:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203277.828923:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187811840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203278.532467:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203279.556474:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203280.580498:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203281.604491:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203282.628470:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203282.948424:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203282.948432:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203282.948436:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203282.948445:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4738 00000100:00080000:2.0:1788203282.948454:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203282.948460:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203282.948493:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187811968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203282.948512:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203282.948515:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203282.948516:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203282.948520:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4738 00000100:00080000:2.0:1788203282.948522:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203282.948525:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203282.948535:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203282.948543:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187811968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203282.948553:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187812096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203282.948572:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203282.948576:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203282.948578:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203282.948582:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4738 00000100:00080000:2.0:1788203282.948587:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203282.948591:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:3.0:1788203282.948596:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:2.0:1788203282.948597:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203282.948612:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187812224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203282.948619:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203282.948621:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203282.948622:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203282.948625:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4738 00000100:00080000:2.0:1788203282.948629:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203282.948640:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203282.948646:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203282.948646:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203282/real 1788203282] req@ffff8e82894e6300 x1875062187811968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203282.948659:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187812352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203282.948668:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203282.948670:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203282.948673:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203282.948675:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187811968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203282.948677:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4738 00000100:00080000:2.0:1788203282.948682:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203282.948686:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203282.948691:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203282.948701:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187812480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203282.948702:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203282.948712:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203282.948717:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187811968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203282.948744:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187812096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203282.948763:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203282.948774:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203282/real 1788203282] req@ffff8e82894e7800 x1875062187812096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203282.948782:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187812096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203282.948788:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203282.948790:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203282.948793:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187812096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203282.948796:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187812224:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203282.948807:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203282.948813:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203282/real 1788203282] req@ffff8e82894e4a80 x1875062187812224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203282.948821:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187812224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203282.948826:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203282.948827:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203282.948829:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187812224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203282.948833:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187812352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203282.948842:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203282.948848:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203282/real 1788203282] req@ffff8e82894e4e00 x1875062187812352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203282.948854:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187812352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203282.948860:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203282.948862:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203282.948864:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187812352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203282.948867:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187812480:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203282.948877:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203282.948882:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203282/real 1788203282] req@ffff8e82894e4000 x1875062187812480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203282.948888:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187812480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203282.948894:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203282.948896:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203282.948912:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187812480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203283.652465:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203284.676526:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203285.700474:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203286.724519:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203287.748471:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203288.068413:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203288.068423:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203288.068427:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203288.068435:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4743 00000100:00080000:2.0:1788203288.068444:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203288.068451:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203288.068483:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187812608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203288.068512:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203288.068515:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203288.068517:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203288.068520:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4743 00000100:00080000:2.0:1788203288.068522:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203288.068525:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203288.068532:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203288.068541:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187812736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203288.068549:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203288.068554:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203288.068555:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203288.068563:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4743 00000100:00080000:2.0:1788203288.068567:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203288.068573:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203288.068579:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203288.068593:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187812864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203288.068609:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187812608:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203288.068611:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203288.068627:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203288.068629:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203288.068634:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4743 00000100:00080000:2.0:1788203288.068639:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203288.068644:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203288.068652:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203288.068668:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187812992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203288.068676:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203288.068680:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203288.068681:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203288.068685:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4743 00000100:00080000:2.0:1788203288.068688:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203288.068691:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203288.068698:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788203288.068710:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203288.068711:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187813120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203288.068771:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203288/real 1788203288] req@ffff8e82894e7480 x1875062187812608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203288.068793:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187812608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203288.068805:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203288.068821:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203288.068826:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187812608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203288.068836:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187812736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203288.068863:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203288.068874:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203288/real 1788203288] req@ffff8e82894e6680 x1875062187812736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203288.068884:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187812736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203288.068891:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203288.068894:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203288.068897:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187812736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203288.068921:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187812864:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203288.068938:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203288.068951:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203288/real 1788203288] req@ffff8e82894e5880 x1875062187812864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203288.068970:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187812864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203288.068987:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203288.068991:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203288.068994:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187812864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203288.068999:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187812992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203288.069020:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203288.069034:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203288/real 1788203288] req@ffff8e82894e6a00 x1875062187812992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203288.069043:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187812992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203288.069049:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203288.069051:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203288.069054:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187812992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203288.069061:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187813120:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203288.069088:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203288.069107:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203288/real 1788203288] req@ffff8e82894e4000 x1875062187813120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203313 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203288.069122:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187813120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203313 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203288.069129:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203288.069131:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203288.069135:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187813120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203288.772487:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203289.796506:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203290.820486:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203291.844488:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203292.868507:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203293.188387:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203293.188395:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203293.188399:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203293.188407:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4748 00000100:00080000:2.0:1788203293.188416:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203293.188422:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203293.188453:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187813248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203293.188479:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203293.188484:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203293.188486:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203293.188491:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4748 00000100:00080000:2.0:1788203293.188494:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203293.188499:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203293.188522:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203293.188539:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187813376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203293.188550:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788203293.188553:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187813248:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203293.188555:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203293.188559:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203293.188567:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4748 00000100:00080000:2.0:1788203293.188571:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203293.188576:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203293.188581:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203293.188596:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187813504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203293.188608:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203293.188613:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203293.188615:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203293.188619:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4748 00000100:00080000:2.0:1788203293.188623:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788203293.188628:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203293.188633:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203293.188640:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203293.188652:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187813632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203293.188663:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203293.188666:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203293.188668:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203293.188673:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4748 00000100:00080000:2.0:1788203293.188677:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203293.188682:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203293.188687:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203293/real 1788203293] req@ffff8e82894e4e00 x1875062187813248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788203293.188691:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203293.188715:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187813248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203293.188716:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187813760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203293.188749:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203293.188758:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203293.188764:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187813248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203293.188787:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187813376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203293.188806:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203293.188815:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203293/real 1788203293] req@ffff8e82894e4a80 x1875062187813376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203293.188824:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187813376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203293.188829:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203293.188832:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203293.188834:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187813376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203293.188837:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187813504:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203293.188848:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203293.188855:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203293/real 1788203293] req@ffff8e82894e7800 x1875062187813504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203293.188863:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187813504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203293.188868:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203293.188870:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203293.188872:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187813504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203293.188875:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187813632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203293.188885:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203293.188891:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203293/real 1788203293] req@ffff8e82894e6300 x1875062187813632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203293.188898:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187813632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203293.188903:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203293.188905:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203293.188907:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187813632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203293.188910:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187813760:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203293.188919:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203293.188925:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203293/real 1788203293] req@ffff8e82894e7100 x1875062187813760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203318 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203293.188941:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187813760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203318 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203293.188949:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203293.188952:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203293.188956:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187813760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203293.892485:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203294.916497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203295.940480:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203296.964492:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203297.988445:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203298.308425:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203298.308436:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203298.308441:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203298.308452:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4753 00000100:00080000:0.0:1788203298.308462:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203298.308469:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203298.308505:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187813888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203298.308531:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203298.308536:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203298.308538:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203298.308542:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4753 00000100:00080000:0.0:1788203298.308546:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203298.308550:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203298.308559:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203298.308573:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187814016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203298.308585:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187813888:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203298.308586:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203298.308596:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203298.308599:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203298.308604:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4753 00000100:00080000:0.0:1788203298.308607:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203298.308611:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203298.308616:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203298.308628:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187814144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203298.308659:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203298.308662:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203298.308664:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203298.308671:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4753 00000800:00000100:1.0:1788203298.308674:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203298.308677:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203298.308684:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203298.308690:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203298.308703:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187814272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203298.308713:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203298.308716:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203298.308718:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203298.308725:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4753 00000100:00100000:1.0:1788203298.308725:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203298/real 1788203298] req@ffff8e828793f480 x1875062187813888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203298.308732:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203298.308740:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203298.308748:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203298.308756:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187813888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203298.308766:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203298.308769:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203298.308774:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187814400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203298.308775:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187813888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203298.308803:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187814016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203298.308837:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203298.308848:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203298/real 1788203298] req@ffff8e828793dc00 x1875062187814016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203298.308856:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187814016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203298.308862:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203298.308868:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203298.308870:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187814016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203298.308875:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187814144:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203298.308888:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203298.308895:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203298/real 1788203298] req@ffff8e828793c000 x1875062187814144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203298.308902:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187814144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203298.308907:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203298.308909:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203298.308911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187814144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203298.308915:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187814272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203298.308926:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203298.308932:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203298/real 1788203298] req@ffff8e828793ca80 x1875062187814272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203298.308939:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187814272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203298.308943:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203298.308945:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203298.308948:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187814272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203298.308951:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187814400:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203298.308961:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203298.308966:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203298/real 1788203298] req@ffff8e828793c700 x1875062187814400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203298.308973:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187814400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203298.308978:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203298.308980:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203298.308981:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187814400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203299.012468:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203300.036572:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203301.060498:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203302.084490:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203303.108488:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203303.428383:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203303.428392:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203303.428396:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203303.428405:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4758 00000100:00080000:0.0:1788203303.428414:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203303.428421:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203303.428453:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187814528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203303.428475:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203303.428478:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203303.428479:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203303.428482:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4758 00000100:00080000:0.0:1788203303.428484:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203303.428487:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203303.428494:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203303.428501:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187814656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203303.428509:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203303.428511:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203303.428512:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203303.428515:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4758 00000100:00080000:0.0:1788203303.428517:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203303.428519:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203303.428522:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203303.428530:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187814784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203303.428561:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203303.428565:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203303.428567:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203303.428574:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4758 00000100:00080000:0.0:1788203303.428580:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788203303.428580:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187814528:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203303.428592:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203303.428601:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203303.428613:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187814912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203303.428620:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203303.428622:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203303.428623:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203303.428626:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4758 00000100:00080000:0.0:1788203303.428629:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203303.428631:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203303.428633:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203303.428642:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187815040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203303.428742:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187814656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203303.428805:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187814784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203303.428841:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187814912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203303.428870:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187815040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203304.132455:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203305.156494:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203306.180534:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203306.500627:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788203306.500689:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788203306.500708:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203306.500766:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203306.500780:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203306.500793:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203306.500807:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788203306.500809:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203303/real 1788203306] req@ffff8e828793e680 x1875062187814784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203306.500842:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187814784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203306.500857:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203306.500863:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203306.500868:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187814784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203306.500877:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203303/real 1788203306] req@ffff8e828793f100 x1875062187814912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203306.500891:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187814912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203306.500898:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203306.500902:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203306.500905:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187814912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203306.500913:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203303/real 1788203306] req@ffff8e828793c700 x1875062187815040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203306.500926:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187815040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203306.500935:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203306.500939:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203306.500942:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187815040:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203306.500950:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203303/real 1788203306] req@ffff8e828793ce00 x1875062187814528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203306.500965:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187814528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203306.500973:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203306.500977:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203306.500980:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187814528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203306.500988:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203303/real 1788203306] req@ffff8e828793f800 x1875062187814656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203306.501002:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187814656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203306.501010:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203306.501014:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203306.501017:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187814656:192.168.201.121@tcp:38:kworker.0 00010000:00010000:0.0:1788203307.204459:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203308.228521:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203308.548401:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203308.548409:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203308.548413:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203308.548423:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4763 00000100:00080000:0.0:1788203308.548438:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203308.548444:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203308.548491:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187815168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203308.548513:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203308.548516:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203308.548518:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203308.548521:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4763 00000100:00080000:0.0:1788203308.548524:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203308.548526:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203308.548534:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203308.548549:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187815296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203308.548557:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203308.548560:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203308.548561:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203308.548564:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4763 00000100:00080000:0.0:1788203308.548566:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203308.548568:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203308.548571:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203308.548580:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187815424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203308.548585:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187815168:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203308.548609:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203308.548613:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203308.548615:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203308.548621:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4763 00000100:00080000:0.0:1788203308.548625:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203308.548630:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203308.548637:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203308.548652:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187815552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203308.548664:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203308.548667:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203308.548669:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203308.548673:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4763 00000100:00080000:0.0:1788203308.548682:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203308.548688:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203308.548696:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788203308.548703:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203308.548723:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187815680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203308.548827:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203308/real 1788203308] req@ffff8e828793ca80 x1875062187815168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203308.548857:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187815168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203308.548878:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203308.548885:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203308.548892:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187815168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203308.548903:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187815296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203308.548937:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203308.548954:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203308/real 1788203308] req@ffff8e828793c000 x1875062187815296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203308.548969:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187815296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203308.548980:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203308.548984:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203308.548988:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187815296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203308.549030:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187815424:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203308.549054:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203308.549066:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203308/real 1788203308] req@ffff8e828793dc00 x1875062187815424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203308.549079:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187815424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203308.549089:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203308.549093:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203308.549097:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187815424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203308.549104:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187815552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203308.549125:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203308.549137:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203308/real 1788203308] req@ffff8e828793f480 x1875062187815552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203308.549150:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187815552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203308.549159:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203308.549163:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203308.549167:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187815552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203308.549174:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187815680:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203308.549196:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203308.549208:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203308/real 1788203308] req@ffff8e828793c380 x1875062187815680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203308.549222:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187815680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203308.549232:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203308.549236:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203308.549241:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187815680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203309.253440:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203310.276467:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203311.300467:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203312.324523:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203313.348459:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203313.668410:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203313.668418:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203313.668422:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203313.668430:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4768 00000100:00080000:0.0:1788203313.668439:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203313.668445:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203313.668477:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187815808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203313.668501:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203313.668504:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203313.668505:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203313.668509:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4768 00000100:00080000:0.0:1788203313.668512:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203313.668515:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203313.668522:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203313.668532:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187815936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203313.668541:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203313.668543:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203313.668545:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203313.668549:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4768 00000100:00100000:1.0:1788203313.668551:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187815808:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203313.668552:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203313.668558:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203313.668562:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203313.668569:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187816064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203313.668580:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203313.668582:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203313.668584:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203313.668587:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4768 00000100:00080000:0.0:1788203313.668589:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203313.668591:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203313.668595:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203313.668601:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187816192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203313.668607:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203313.668609:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203313.668610:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203313.668612:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4768 00000100:00080000:0.0:1788203313.668615:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203313.668620:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:1.0:1788203313.668622:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788203313.668624:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203313.668639:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187816320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203313.668669:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203313/real 1788203313] req@ffff8e828793d880 x1875062187815808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203313.668697:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187815808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203313.668707:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203313.668710:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203313.668713:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187815808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203313.668736:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187815936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203313.668762:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203313.668775:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203313/real 1788203313] req@ffff8e828793fb80 x1875062187815936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203313.668784:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187815936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203313.668789:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203313.668792:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203313.668794:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187815936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203313.668798:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187816064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203313.668811:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203313.668818:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203313/real 1788203313] req@ffff8e828793d500 x1875062187816064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203313.668825:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187816064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203313.668834:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203313.668836:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203313.668839:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187816064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203313.668842:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187816192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203313.668852:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203313.668858:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203313/real 1788203313] req@ffff8e828793e300 x1875062187816192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203313.668865:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187816192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203313.668870:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203313.668872:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203313.668874:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187816192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203313.668878:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187816320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203313.668887:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203313.668893:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203313/real 1788203313] req@ffff8e828793c380 x1875062187816320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203313.668899:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187816320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203313.668904:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203313.668906:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203313.668909:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187816320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203314.372435:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203315.396448:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203316.420497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203317.444440:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203318.468498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203318.788447:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203318.788455:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203318.788459:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203318.788468:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4773 00000100:00080000:2.0:1788203318.788477:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203318.788489:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203318.788530:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187816448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203318.788555:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203318.788558:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203318.788565:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203318.788568:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4773 00000100:00080000:2.0:1788203318.788572:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203318.788576:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203318.788584:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203318.788594:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187816576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203318.788602:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203318.788604:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203318.788606:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203318.788611:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4773 00000100:00100000:1.0:1788203318.788613:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187816448:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203318.788613:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203318.788618:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203318.788626:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203318.788639:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187816704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203318.788650:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203318.788654:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203318.788656:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203318.788660:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4773 00000100:00080000:2.0:1788203318.788664:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203318.788669:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203318.788675:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788203318.788689:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203318.788689:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187816832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203318.788699:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203318.788703:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203318.788704:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203318.788708:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4773 00000100:00080000:2.0:1788203318.788712:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203318.788714:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203318.788718:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203318.788725:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187816960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203318.788739:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203318/real 1788203318] req@ffff8e82894e5c00 x1875062187816448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203318.788757:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187816448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203318.788765:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203318.788768:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203318.788771:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187816448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203318.788790:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187816576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203318.788808:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203318.788816:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203318/real 1788203318] req@ffff8e82894e4380 x1875062187816576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203318.788824:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187816576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203318.788829:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203318.788832:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203318.788834:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187816576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203318.788838:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187816704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203318.788848:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203318.788854:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203318/real 1788203318] req@ffff8e82894e5f80 x1875062187816704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203318.788862:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187816704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203318.788870:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203318.788873:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203318.788877:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187816704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203318.788883:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187816832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203318.788903:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203318.788915:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203318/real 1788203318] req@ffff8e82894e6d80 x1875062187816832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203318.788927:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187816832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203318.788935:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203318.788938:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203318.788941:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187816832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203318.788947:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187816960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203318.788964:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203318.788975:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203318/real 1788203318] req@ffff8e82894e7100 x1875062187816960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203318.788985:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187816960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203318.788993:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203318.788996:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203318.788999:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187816960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203319.492483:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203320.516487:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203321.540486:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203322.564477:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203323.588495:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203323.908377:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203323.908385:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203323.908390:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203323.908399:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4778 00000100:00080000:2.0:1788203323.908408:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203323.908414:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203323.908448:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187817088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203323.908472:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203323.908475:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203323.908476:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203323.908479:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4778 00000100:00080000:2.0:1788203323.908481:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203323.908484:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203323.908491:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203323.908498:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187817216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203323.908506:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203323.908509:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203323.908510:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203323.908514:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4778 00000100:00080000:2.0:1788203323.908516:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203323.908518:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203323.908521:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203323.908528:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187817088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203323.908531:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187817344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203323.908545:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203323.908547:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203323.908549:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203323.908551:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4778 00000100:00080000:2.0:1788203323.908553:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203323.908555:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203323.908558:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203323.908566:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187817472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203323.908572:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203323.908574:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203323.908575:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203323.908579:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4778 00000100:00080000:2.0:1788203323.908582:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203323.908585:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203323.908589:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788203323.908594:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203323.908601:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187817600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203323.908643:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203323/real 1788203323] req@ffff8e82894e6300 x1875062187817088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203323.908664:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187817088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203323.908672:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203323.908675:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203323.908678:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187817088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203323.908684:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187817216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203323.908706:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203323.908718:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203323/real 1788203323] req@ffff8e82894e7800 x1875062187817216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203323.908734:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187817216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203323.908741:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203323.908744:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203323.908748:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187817216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203323.908768:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187817344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203323.908787:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203323.908801:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203323/real 1788203323] req@ffff8e82894e4a80 x1875062187817344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203323.908816:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187817344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203323.908826:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203323.908830:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203323.908833:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187817344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203323.908839:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187817472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203323.908856:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203323.908867:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203323/real 1788203323] req@ffff8e82894e4e00 x1875062187817472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203323.908877:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187817472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203323.908888:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203323.908891:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203323.908894:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187817472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203323.908899:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187817600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203323.908913:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203323.908924:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203323/real 1788203323] req@ffff8e82894e4000 x1875062187817600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203323.908933:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187817600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203323.908939:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203323.908941:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203323.908945:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187817600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203324.612479:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203325.636516:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203326.660463:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203327.684495:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203328.708460:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203329.028416:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203329.028426:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203329.028430:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203329.028440:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4784 00000100:00080000:2.0:1788203329.028453:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203329.028460:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203329.028498:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187817728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203329.028525:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203329.028527:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203329.028529:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203329.028532:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4784 00000100:00080000:2.0:1788203329.028540:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203329.028544:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203329.028551:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203329.028561:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187817856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203329.028570:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203329.028572:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203329.028576:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788203329.028577:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187817728:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203329.028581:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4784 00000100:00080000:2.0:1788203329.028588:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203329.028592:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203329.028597:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203329.028611:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187817984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203329.028622:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203329.028626:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203329.028628:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203329.028632:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4784 00000100:00080000:2.0:1788203329.028636:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788203329.028637:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203329.028643:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203329.028648:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203329.028658:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187818112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203329.028665:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203329.028668:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203329.028669:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203329.028672:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4784 00000100:00080000:2.0:1788203329.028675:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788203329.028679:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203329/real 1788203329] req@ffff8e82894e6a00 x1875062187817728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203329.028681:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203329.028687:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203329.028705:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187817728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203329.028705:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187818240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203329.028738:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203329.028743:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203329.028747:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187817728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203329.028766:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187817856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203329.028780:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203329.028789:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203329/real 1788203329] req@ffff8e82894e5880 x1875062187817856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203329.028797:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187817856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203329.028802:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203329.028804:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203329.028806:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187817856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203329.028810:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187817984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203329.028820:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203329.028825:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203329/real 1788203329] req@ffff8e82894e6680 x1875062187817984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203329.028833:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187817984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203329.028838:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203329.028839:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203329.028841:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187817984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203329.028845:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187818112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203329.028855:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203329.028860:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203329/real 1788203329] req@ffff8e82894e7480 x1875062187818112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203329.028867:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187818112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203329.028872:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203329.028874:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203329.028876:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187818112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203329.028879:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187818240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203329.028888:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203329.028893:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203329/real 1788203329] req@ffff8e82894e4000 x1875062187818240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203329.028900:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187818240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203354 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203329.028905:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203329.028906:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203329.028908:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187818240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203329.732487:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203330.756458:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203331.780484:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203332.804455:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203333.828508:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203334.148408:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203334.148416:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203334.148420:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203334.148428:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4789 00000100:00080000:1.0:1788203334.148438:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203334.148443:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203334.148475:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187818368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203334.148499:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203334.148502:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203334.148503:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203334.148506:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4789 00000100:00080000:1.0:1788203334.148509:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203334.148512:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203334.148520:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203334.148530:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187818368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203334.148532:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187818496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203334.148544:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203334.148547:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203334.148548:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203334.148551:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4789 00000100:00080000:1.0:1788203334.148553:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203334.148555:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203334.148560:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203334.148569:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187818624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203334.148576:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203334.148578:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203334.148580:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203334.148583:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4789 00000100:00080000:1.0:1788203334.148586:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203334.148588:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203334.148592:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203334.148598:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187818752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203334.148604:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788203334.148613:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203334.148616:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203334.148617:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203334.148621:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4789 00000100:00080000:1.0:1788203334.148625:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203334.148627:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203334.148629:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203334.148637:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187818880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203334.148646:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203334/real 1788203334] req@ffff8e828d720a80 x1875062187818368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203334.148662:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187818368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203334.148674:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203334.148677:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203334.148680:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187818368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203334.148699:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187818496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203334.148715:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203334.148723:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203334/real 1788203334] req@ffff8e828d721180 x1875062187818496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203334.148731:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187818496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203334.148737:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203334.148739:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203334.148741:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187818496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203334.148744:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187818624:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203334.148755:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203334.148761:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203334/real 1788203334] req@ffff8e828d722d80 x1875062187818624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203334.148768:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187818624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203334.148773:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203334.148774:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203334.148776:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187818624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203334.148780:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187818752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203334.148791:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203334.148796:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203334/real 1788203334] req@ffff8e828d722300 x1875062187818752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203334.148804:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187818752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203334.148809:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203334.148810:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203334.148812:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187818752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203334.148816:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187818880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203334.148824:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203334.148830:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203334/real 1788203334] req@ffff8e828d720380 x1875062187818880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203334.148836:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187818880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203359 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203334.148840:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203334.148842:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203334.148844:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187818880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203334.852433:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203335.876512:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203336.900535:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203337.924538:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203338.948570:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203339.268468:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203339.268481:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203339.268486:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203339.268499:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4794 00000100:00080000:0.0:1788203339.268513:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203339.268521:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203339.268571:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187819008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203339.268599:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203339.268602:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203339.268604:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203339.268607:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4794 00000100:00080000:0.0:1788203339.268610:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203339.268613:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203339.268622:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203339.268637:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187819136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203339.268650:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203339.268654:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203339.268664:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203339.268670:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4794 00000100:00080000:0.0:1788203339.268674:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203339.268683:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203339.268691:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187819008:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788203339.268694:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203339.268717:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187819264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203339.268729:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203339.268732:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203339.268733:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203339.268737:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4794 00000100:00080000:0.0:1788203339.268740:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203339.268744:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203339.268749:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203339.268757:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187819392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203339.268765:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203339.268768:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203339.268769:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203339.268772:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4794 00000100:00080000:0.0:1788203339.268777:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203339.268780:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203339.268783:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788203339.268791:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203339.268798:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187819520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203339.268862:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203339/real 1788203339] req@ffff8e828793f480 x1875062187819008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203339.268886:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187819008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203339.268899:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203339.268904:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203339.268908:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187819008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203339.268917:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187819136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203339.268973:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203339.268987:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203339/real 1788203339] req@ffff8e828793dc00 x1875062187819136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203339.269001:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187819136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203339.269010:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203339.269014:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203339.269018:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187819136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203339.269044:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187819264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203339.269061:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203339.269070:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203339/real 1788203339] req@ffff8e828793c000 x1875062187819264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203339.269077:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187819264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203339.269087:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203339.269090:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203339.269092:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187819264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203339.269096:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187819392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203339.269107:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203339.269113:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203339/real 1788203339] req@ffff8e828793ca80 x1875062187819392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203339.269120:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187819392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203339.269125:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203339.269127:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203339.269129:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187819392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203339.269132:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187819520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203339.269142:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203339.269148:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203339/real 1788203339] req@ffff8e828793f800 x1875062187819520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203364 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203339.269155:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187819520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203364 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203339.269159:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203339.269161:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203339.269163:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187819520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203339.972485:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203340.996523:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203342.020540:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203343.044468:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203344.068532:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203344.388410:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203344.388419:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203344.388423:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203344.388431:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4799 00000100:00080000:0.0:1788203344.388440:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203344.388448:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203344.388487:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187819648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203344.388519:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203344.388523:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203344.388525:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203344.388530:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4799 00000100:00080000:0.0:1788203344.388534:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203344.388539:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203344.388547:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203344.388559:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187819776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203344.388568:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203344.388573:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788203344.388573:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187819648:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203344.388578:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203344.388585:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4799 00000100:00080000:0.0:1788203344.388588:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203344.388590:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203344.388594:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203344.388603:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187819904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203344.388614:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203344.388617:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203344.388618:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203344.388621:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4799 00000100:00080000:0.0:1788203344.388625:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203344.388628:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203344.388631:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788203344.388635:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203344.388642:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187820032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203344.388650:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203344.388653:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203344.388654:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203344.388657:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4799 00000100:00080000:0.0:1788203344.388659:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203344.388662:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203344.388665:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203344.388674:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203344/real 1788203344] req@ffff8e828793ce00 x1875062187819648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203344.388676:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187820160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203344.388709:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187819648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203344.388718:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203344.388722:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203344.388726:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187819648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203344.388744:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187819776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203344.388758:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203344.388766:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203344/real 1788203344] req@ffff8e828793c700 x1875062187819776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203344.388774:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187819776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203344.388780:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203344.388782:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203344.388784:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187819776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203344.388787:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187819904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203344.388798:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203344.388804:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203344/real 1788203344] req@ffff8e828793f100 x1875062187819904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203344.388812:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187819904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203344.388817:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203344.388818:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203344.388820:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187819904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203344.388823:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187820032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203344.388852:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203344.388861:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203344/real 1788203344] req@ffff8e828793e680 x1875062187820032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203344.388869:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187820032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203344.388874:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203344.388876:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203344.388878:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187820032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203344.388882:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187820160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203344.388893:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203344.388899:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203344/real 1788203344] req@ffff8e828793f800 x1875062187820160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203344.388906:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187820160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203344.388911:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203344.388913:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203344.388915:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187820160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203345.092489:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203346.116472:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203347.140435:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203348.164448:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203349.188513:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203349.508464:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203349.508472:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203349.508476:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203349.508484:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4804 00000100:00080000:2.0:1788203349.508493:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203349.508500:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203349.508537:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187820288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203349.508572:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203349.508575:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203349.508577:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203349.508581:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4804 00000100:00080000:2.0:1788203349.508584:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203349.508587:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203349.508595:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203349.508606:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187820416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203349.508614:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203349.508616:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203349.508618:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203349.508620:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4804 00000100:00080000:2.0:1788203349.508623:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203349.508628:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203349.508632:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187820288:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788203349.508633:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203349.508642:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187820544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203349.508650:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203349.508652:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203349.508653:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203349.508656:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4804 00000100:00080000:2.0:1788203349.508659:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203349.508663:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203349.508668:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203349.508676:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187820672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203349.508683:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203349.508685:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203349.508687:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203349.508690:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4804 00000100:00080000:2.0:1788203349.508692:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203349.508695:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203349.508698:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788203349.508702:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203349.508709:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187820800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203349.508762:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203349/real 1788203349] req@ffff8e82894e4e00 x1875062187820288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203349.508781:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187820288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203349.508793:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203349.508797:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203349.508801:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187820288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203349.508809:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187820416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203349.508832:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203349.508845:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203349/real 1788203349] req@ffff8e82894e4a80 x1875062187820416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203349.508859:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187820416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203349.508868:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203349.508871:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203349.508876:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187820416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203349.508899:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187820544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203349.508919:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203349.508930:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203349/real 1788203349] req@ffff8e82894e7800 x1875062187820544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203349.508943:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187820544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203349.508952:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203349.508955:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203349.508959:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187820544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203349.508966:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187820672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203349.508984:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203349.508996:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203349/real 1788203349] req@ffff8e82894e6300 x1875062187820672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203349.509021:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187820672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203349.509030:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203349.509033:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203349.509037:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187820672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203349.509044:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187820800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203349.509063:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203349.509074:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203349/real 1788203349] req@ffff8e82894e7100 x1875062187820800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203349.509087:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187820800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203349.509095:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203349.509098:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203349.509102:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187820800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203350.212493:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203351.236482:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203352.260498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203353.284540:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203354.308454:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203354.628438:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203354.628446:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203354.628450:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203354.628459:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4809 00000100:00080000:0.0:1788203354.628468:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203354.628474:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203354.628508:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187820928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203354.628536:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203354.628540:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203354.628543:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203354.628548:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4809 00000100:00080000:0.0:1788203354.628552:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203354.628556:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203354.628567:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203354.628581:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187821056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203354.628586:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187820928:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203354.628592:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203354.628596:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203354.628597:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203354.628600:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4809 00000100:00080000:0.0:1788203354.628603:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203354.628606:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203354.628609:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203354.628618:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187821184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203354.628630:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203354.628632:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203354.628634:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203354.628636:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4809 00000100:00080000:0.0:1788203354.628638:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203354.628642:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:1.0:1788203354.628646:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788203354.628648:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203354.628684:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187821312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203354.628692:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203354.628694:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203354.628695:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203354.628699:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4809 00000100:00080000:0.0:1788203354.628702:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203354.628708:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203354.628709:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203354/real 1788203354] req@ffff8e828793ca80 x1875062187820928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788203354.628712:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203354.628724:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187821440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203354.628736:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187820928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203354.628748:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203354.628752:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203354.628757:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187820928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203354.628780:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187821056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203354.628802:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203354.628815:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203354/real 1788203354] req@ffff8e828793c000 x1875062187821056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203354.628828:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187821056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203354.628836:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203354.628840:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203354.628843:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187821056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203354.628850:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187821184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203354.628886:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203354.628904:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203354/real 1788203354] req@ffff8e828793dc00 x1875062187821184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203354.628916:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187821184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203354.628924:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203354.628927:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203354.628930:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187821184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203354.628937:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187821312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203354.628967:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203354.628979:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203354/real 1788203354] req@ffff8e828793f480 x1875062187821312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203354.628991:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187821312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203354.629004:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203354.629007:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203354.629010:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187821312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203354.629015:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187821440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203354.629033:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203354.629043:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203354/real 1788203354] req@ffff8e828793c380 x1875062187821440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203354.629056:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187821440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203354.629064:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203354.629068:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203354.629071:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187821440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203355.332451:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203356.356458:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203357.380481:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203358.404445:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203359.428505:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203359.748415:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203359.748424:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203359.748429:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203359.748438:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4814 00000100:00080000:0.0:1788203359.748447:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203359.748453:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203359.748490:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187821568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203359.748512:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203359.748514:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203359.748516:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203359.748519:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4814 00000100:00080000:0.0:1788203359.748521:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203359.748524:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203359.748532:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203359.748543:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187821696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203359.748551:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203359.748554:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203359.748557:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788203359.748558:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187821568:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203359.748561:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4814 00000100:00080000:0.0:1788203359.748582:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203359.748586:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203359.748591:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203359.748603:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187821824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203359.748615:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203359.748618:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203359.748619:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203359.748622:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4814 00000100:00080000:0.0:1788203359.748625:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203359.748628:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203359.748632:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788203359.748637:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203359.748644:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187821952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203359.748652:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203359.748654:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203359.748656:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203359.748659:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4814 00000100:00080000:0.0:1788203359.748661:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203359.748664:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203359.748669:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203359.748676:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203359/real 1788203359] req@ffff8e828793e300 x1875062187821568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203359.748680:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187822080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203359.748715:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187821568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203359.748730:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203359.748733:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203359.748736:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187821568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203359.748760:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187821696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203359.748780:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203359.748792:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203359/real 1788203359] req@ffff8e828793d500 x1875062187821696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203359.748802:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187821696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203359.748808:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203359.748811:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203359.748814:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187821696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203359.748820:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187821824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203359.748837:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203359.748848:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203359/real 1788203359] req@ffff8e828793fb80 x1875062187821824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203359.748861:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187821824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203359.748870:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203359.748874:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203359.748877:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187821824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203359.748883:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187821952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203359.748904:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203359.748917:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203359/real 1788203359] req@ffff8e828793d880 x1875062187821952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203359.748937:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187821952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203359.748946:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203359.748949:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203359.748951:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187821952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203359.748957:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187822080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203359.748975:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203359.748983:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203359/real 1788203359] req@ffff8e828793c380 x1875062187822080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203359.748991:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187822080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203359.748996:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203359.748999:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203359.749001:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187822080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203360.452487:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203361.476496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203362.500515:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203363.524539:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203364.548574:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203364.868442:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203364.868454:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203364.868460:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203364.868474:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4819 00000100:00080000:0.0:1788203364.868502:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203364.868511:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203364.868559:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187822208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203364.868588:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203364.868591:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203364.868593:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203364.868597:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4819 00000100:00080000:0.0:1788203364.868599:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203364.868602:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203364.868612:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203364.868625:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187822336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203364.868636:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203364.868640:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203364.868642:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203364.868646:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4819 00000100:00100000:1.0:1788203364.868651:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187822208:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203364.868653:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203364.868663:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203364.868669:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203364.868683:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187822464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203364.868701:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203364.868705:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203364.868707:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203364.868724:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4819 00000100:00080000:0.0:1788203364.868729:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203364.868736:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203364.868742:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788203364.868742:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203364.868757:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187822592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203364.868772:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203364.868775:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203364.868776:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203364.868779:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4819 00000100:00080000:0.0:1788203364.868782:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203364.868785:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203364.868790:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203364.868802:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187822720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203364.868813:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203364/real 1788203364] req@ffff8e828793f480 x1875062187822208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203364.868839:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187822208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203364.868852:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203364.868855:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203364.868859:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187822208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203364.868865:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187822336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203364.868884:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203364.868894:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203364/real 1788203364] req@ffff8e828793dc00 x1875062187822336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203364.868902:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187822336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203364.868907:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203364.868910:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203364.868912:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187822336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203364.868941:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187822464:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203364.868962:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203364.868975:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203364/real 1788203364] req@ffff8e828793c000 x1875062187822464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203364.868989:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187822464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203364.868995:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203364.868997:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203364.868999:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187822464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203364.869003:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187822592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203364.869018:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203364.869025:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203364/real 1788203364] req@ffff8e828793ca80 x1875062187822592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203364.869032:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187822592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203364.869037:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203364.869039:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203364.869041:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187822592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203364.869044:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187822720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203364.869053:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203364.869059:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203364/real 1788203364] req@ffff8e828793f800 x1875062187822720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203364.869066:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187822720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203364.869076:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203364.869078:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203364.869080:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187822720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203365.572575:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203366.596579:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203367.620475:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203368.644496:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203369.668569:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203369.988480:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203369.988496:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203369.988501:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203369.988516:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4825 00000100:00080000:0.0:1788203369.988540:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203369.988551:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203369.988608:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187822848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203369.988645:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203369.988658:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203369.988660:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203369.988665:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4825 00000100:00080000:0.0:1788203369.988670:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203369.988676:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203369.988688:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203369.988705:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187822976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203369.988720:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187822848:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203369.988722:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203369.988730:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203369.988734:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203369.988740:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4825 00000100:00080000:0.0:1788203369.988743:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203369.988747:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203369.988752:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203369.988763:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187823104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203369.988779:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203369.988782:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203369.988783:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203369.988788:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4825 00000100:00080000:0.0:1788203369.988804:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203369.988808:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203369.988813:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203369.988820:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187823232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203369.988828:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203369.988831:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203369.988832:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203369.988835:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4825 00000100:00080000:0.0:1788203369.988837:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203369.988840:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203369.988845:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203369.988857:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187823360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203369.988878:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187822976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203369.988921:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187823104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203369.988952:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187823232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203369.988984:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187823360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203370.692594:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203371.716498:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203372.740591:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203373.060585:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788203373.060628:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788203373.060646:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203373.060700:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203373.060710:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203373.060721:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203373.060731:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788203373.060746:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203369/real 1788203373] req@ffff8e828793f100 x1875062187822976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203373.060772:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187822976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203373.060783:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203373.060788:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203373.060793:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187822976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203373.060801:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203369/real 1788203373] req@ffff8e828793c700 x1875062187823104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203373.060814:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187823104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203373.060822:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203373.060826:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203373.060829:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187823104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203373.060836:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203369/real 1788203373] req@ffff8e828793ce00 x1875062187823232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203373.060848:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187823232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203373.060856:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203373.060859:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203373.060862:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187823232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203373.060869:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203369/real 1788203373] req@ffff8e828793f800 x1875062187823360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203373.060879:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187823360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203373.060886:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203373.060889:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203373.060901:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187823360:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203373.060909:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203369/real 1788203373] req@ffff8e828793e680 x1875062187822848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203373.060923:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187822848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203373.060930:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203373.060933:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203373.060937:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187822848:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788203373.764488:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203374.788526:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203375.108417:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203375.108425:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203375.108428:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203375.108437:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4830 00000100:00080000:0.0:1788203375.108447:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203375.108453:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203375.108484:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187823488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203375.108510:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203375.108513:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203375.108514:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203375.108517:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4830 00000100:00080000:0.0:1788203375.108519:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203375.108522:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203375.108529:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203375.108539:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187823616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203375.108546:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203375.108549:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203375.108551:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203375.108556:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4830 00000100:00080000:0.0:1788203375.108560:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203375.108564:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203375.108569:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203375.108580:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187823744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203375.108591:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203375.108595:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203375.108599:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788203375.108600:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187823488:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203375.108606:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4830 00000100:00080000:0.0:1788203375.108613:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203375.108617:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203375.108623:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203375.108637:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187823872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203375.108647:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203375.108650:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203375.108652:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203375.108656:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4830 00000100:00080000:0.0:1788203375.108661:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203375.108665:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203375.108671:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788203375.108671:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203375.108687:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187824000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203375.108723:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203375/real 1788203375] req@ffff8e828793ca80 x1875062187823488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203375.108742:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187823488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203375.108754:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203375.108758:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203375.108762:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187823488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203375.108770:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187823616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203375.108793:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203375.108823:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203375/real 1788203375] req@ffff8e828793c000 x1875062187823616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203375.108844:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187823616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203375.108870:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203375.108873:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203375.108877:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187823616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203375.108902:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187823744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203375.108924:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203375.108950:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203375/real 1788203375] req@ffff8e828793dc00 x1875062187823744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203375.108962:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187823744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203375.108971:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203375.108974:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203375.108978:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187823744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203375.108984:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187823872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203375.109007:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203375.109031:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203375/real 1788203375] req@ffff8e828793f480 x1875062187823872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203375.109049:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187823872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203375.109062:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203375.109065:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203375.109068:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187823872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203375.109073:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187824000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203375.109092:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203375.109104:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203375/real 1788203375] req@ffff8e828793c380 x1875062187824000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203375.109116:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187824000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203375.109123:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203375.109127:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203375.109129:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187824000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203375.812505:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203376.836476:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203377.861436:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203378.884424:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203379.908411:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203380.228429:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203380.228438:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203380.228441:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203380.228449:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4835 00000100:00080000:0.0:1788203380.228458:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203380.228464:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203380.228501:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187824128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203380.228524:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203380.228527:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203380.228532:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203380.228535:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4835 00000100:00080000:0.0:1788203380.228538:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203380.228540:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203380.228548:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203380.228557:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187824256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203380.228565:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203380.228567:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203380.228568:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203380.228571:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4835 00000100:00080000:0.0:1788203380.228573:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203380.228575:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203380.228579:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203380.228581:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187824128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203380.228587:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187824384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203380.228596:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203380.228598:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203380.228599:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203380.228601:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4835 00000100:00080000:0.0:1788203380.228604:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203380.228606:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203380.228610:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203380.228618:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187824512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203380.228623:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203380.228625:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203380.228627:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203380.228629:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4835 00000100:00080000:0.0:1788203380.228654:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203380.228658:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203380.228663:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788203380.228665:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203380.228681:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187824640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203380.228712:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203380/real 1788203380] req@ffff8e828793d880 x1875062187824128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203380.228733:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187824128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203380.228746:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203380.228750:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203380.228755:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187824128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203380.228763:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187824256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203380.228786:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203380.228799:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203380/real 1788203380] req@ffff8e828793fb80 x1875062187824256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203380.228812:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187824256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203380.228821:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203380.228825:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203380.228828:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187824256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203380.228852:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187824384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203380.228872:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203380.228884:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203380/real 1788203380] req@ffff8e828793d500 x1875062187824384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203380.228897:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187824384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203380.228905:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203380.228909:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203380.228912:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187824384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203380.228919:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187824512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203380.228938:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203380.228949:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203380/real 1788203380] req@ffff8e828793e300 x1875062187824512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203380.228962:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187824512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203380.228971:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203380.228974:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203380.228978:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187824512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203380.228985:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187824640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203380.229003:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203380.229013:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203380/real 1788203380] req@ffff8e828793c380 x1875062187824640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203380.229025:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187824640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203380.229034:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203380.229038:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203380.229042:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187824640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203380.932910:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203381.956543:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203382.980540:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203384.004543:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203385.028474:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203385.348443:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203385.348451:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203385.348455:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203385.348463:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4840 00000100:00080000:2.0:1788203385.348471:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203385.348478:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203385.348511:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187824768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203385.348542:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203385.348545:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203385.348546:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203385.348549:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4840 00000100:00080000:2.0:1788203385.348551:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203385.348554:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203385.348562:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203385.348575:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187824896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203385.348586:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203385.348590:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203385.348592:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203385.348596:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4840 00000100:00080000:2.0:1788203385.348600:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203385.348605:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203385.348611:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203385.348628:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187825024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203385.348633:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187824768:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203385.348637:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203385.348652:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203385.348658:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203385.348662:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4840 00000100:00080000:2.0:1788203385.348664:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203385.348667:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203385.348671:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203385.348678:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187825152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203385.348685:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203385.348687:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203385.348688:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203385.348691:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4840 00000100:00080000:2.0:1788203385.348693:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203385.348695:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203385.348698:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203385.348705:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187825280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203385.348748:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203385.348816:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203385/real 1788203385] req@ffff8e82894e6d80 x1875062187824768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203385.348838:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187824768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203385.348852:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203385.348857:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203385.348862:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187824768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203385.348871:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187824896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203385.348898:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203385.348912:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203385/real 1788203385] req@ffff8e82894e5f80 x1875062187824896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203385.348925:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187824896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203385.348933:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203385.348936:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203385.348939:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187824896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203385.348965:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187825024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203385.348980:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203385.348987:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203385/real 1788203385] req@ffff8e82894e4380 x1875062187825024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203385.348996:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187825024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203385.349002:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203385.349004:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203385.349007:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187825024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203385.349011:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187825152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203385.349022:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203385.349029:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203385/real 1788203385] req@ffff8e82894e5c00 x1875062187825152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203385.349037:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187825152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203385.349042:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203385.349044:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203385.349045:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187825152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203385.349049:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187825280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203385.349059:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203385.349065:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203385/real 1788203385] req@ffff8e82894e7100 x1875062187825280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203385.349072:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187825280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203385.349082:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203385.349085:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203385.349087:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187825280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203386.052480:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203387.076498:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203388.100508:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203389.124498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203390.148485:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203390.468439:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203390.468448:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203390.468451:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203390.468459:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4845 00000100:00080000:0.0:1788203390.468469:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203390.468475:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203390.468507:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187825408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203390.468531:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203390.468534:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203390.468536:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203390.468538:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4845 00000100:00080000:0.0:1788203390.468541:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203390.468543:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203390.468551:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203390.468559:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187825536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203390.468566:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203390.468568:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203390.468569:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203390.468571:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4845 00000100:00080000:0.0:1788203390.468573:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203390.468575:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203390.468577:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203390.468582:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187825664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203390.468589:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203390.468592:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203390.468593:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203390.468597:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4845 00000100:00080000:0.0:1788203390.468600:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203390.468604:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203390.468610:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203390.468611:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187825408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203390.468628:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187825792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203390.468641:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203390.468645:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203390.468647:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203390.468651:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4845 00000100:00080000:0.0:1788203390.468666:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203390.468670:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203390.468675:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788203390.468675:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203390.468689:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187825920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203390.468714:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203390/real 1788203390] req@ffff8e828793f480 x1875062187825408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203390.468729:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187825408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203390.468737:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203390.468739:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203390.468742:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187825408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203390.468747:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187825536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203390.468760:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203390.468768:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203390/real 1788203390] req@ffff8e828793dc00 x1875062187825536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203390.468776:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187825536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203390.468786:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203390.468788:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203390.468790:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187825536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203390.468793:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187825664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203390.468807:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203390.468817:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203390/real 1788203390] req@ffff8e828793c000 x1875062187825664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203390.468830:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187825664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203390.468839:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203390.468842:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203390.468846:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187825664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203390.468872:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187825792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203390.468894:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203390.468907:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203390/real 1788203390] req@ffff8e828793ca80 x1875062187825792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203390.468920:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187825792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203390.468929:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203390.468932:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203390.468936:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187825792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203390.468943:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187825920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203390.468963:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203390.468975:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203390/real 1788203390] req@ffff8e828793e680 x1875062187825920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203390.468986:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187825920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203390.468994:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203390.468997:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203390.469000:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187825920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203391.172476:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203392.196511:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203393.220507:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203394.244508:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203395.268477:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203395.588417:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203395.588425:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203395.588429:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203395.588438:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4850 00000100:00080000:2.0:1788203395.588447:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203395.588453:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203395.588484:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187826048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203395.588506:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203395.588510:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203395.588512:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203395.588517:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4850 00000100:00080000:2.0:1788203395.588522:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203395.588527:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203395.588538:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203395.588540:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187826048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203395.588559:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187826176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203395.588573:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203395.588576:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203395.588578:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203395.588583:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4850 00000100:00080000:2.0:1788203395.588589:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203395.588594:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:3.0:1788203395.588600:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:2.0:1788203395.588601:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203395.588620:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187826304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203395.588632:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203395.588636:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203395.588639:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203395.588640:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203395/real 1788203395] req@ffff8e82894e6300 x1875062187826048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203395.588645:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4850 00000100:00080000:2.0:1788203395.588650:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203395.588653:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203395.588658:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203395.588668:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187826432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203395.588674:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203395.588676:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203395.588677:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203395.588679:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4850 00000100:00080000:2.0:1788203395.588681:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203395.588683:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203395.588686:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203395.588691:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187826560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203395.588749:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187826048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203395.588762:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203395.588767:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203395.588771:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187826048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203395.588795:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187826176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203395.588934:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203395.588970:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203395/real 1788203395] req@ffff8e82894e7800 x1875062187826176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203395.588987:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187826176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203395.588997:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203395.589002:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203395.589009:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187826176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203395.589018:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187826304:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203395.589081:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203395.589094:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203395/real 1788203395] req@ffff8e82894e4a80 x1875062187826304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203395.589106:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187826304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203395.589114:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203395.589117:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203395.589120:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187826304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203395.589127:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187826432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203395.589146:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203395.589156:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203395/real 1788203395] req@ffff8e82894e4e00 x1875062187826432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203395.589167:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187826432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203395.589175:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203395.589178:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203395.589181:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187826432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203395.589187:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187826560:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203395.589203:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203395.589213:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203395/real 1788203395] req@ffff8e82894e4000 x1875062187826560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203395.589232:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187826560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203395.589239:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203395.589243:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203395.589246:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187826560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203396.292488:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203397.316437:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203398.340532:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203399.364497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203400.388514:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203400.708424:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203400.708435:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203400.708440:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203400.708452:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4855 00000100:00080000:2.0:1788203400.708464:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203400.708471:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203400.708522:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187826688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203400.708554:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203400.708558:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203400.708560:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203400.708565:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4855 00000100:00080000:2.0:1788203400.708570:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203400.708575:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203400.708585:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203400.708588:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187826688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203400.708603:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187826816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203400.708617:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203400.708621:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203400.708622:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203400.708626:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4855 00000100:00080000:2.0:1788203400.708634:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203400.708639:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203400.708645:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203400.708646:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203400.708662:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187826944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203400.708674:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203400.708678:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203400.708681:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203400.708684:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203400/real 1788203400] req@ffff8e82894e7480 x1875062187826688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203400.708688:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4855 00000100:00080000:2.0:1788203400.708696:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203400.708703:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203400.708705:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187826688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788203400.708712:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788203400.708716:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203400.708720:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203400.708723:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187826688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203400.708730:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187827072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203400.708744:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788203400.708744:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187826816:192.168.201.121@tcp:38:kworker.0 00000100:00080000:2.0:1788203400.708752:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203400.708755:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203400.708759:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4855 00000100:00080000:2.0:1788203400.708764:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788203400.708768:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203400.708772:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203400.708779:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203400.708781:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203400/real 1788203400] req@ffff8e82894e6680 x1875062187826816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203400.708792:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187826816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203400.708798:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187827200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203400.708820:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203400.708825:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203400.708828:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187826816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203400.708834:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187826944:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203400.708848:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203400.708872:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203400/real 1788203400] req@ffff8e82894e5880 x1875062187826944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203400.708887:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187826944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203400.708892:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203400.708894:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203400.708896:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187826944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203400.708906:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187827072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203400.708920:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203400.708927:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203400/real 1788203400] req@ffff8e82894e6a00 x1875062187827072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203400.708934:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187827072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203400.708939:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203400.708940:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203400.708943:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187827072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203400.708947:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187827200:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203400.708956:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203400.708961:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203400/real 1788203400] req@ffff8e82894e7480 x1875062187827200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203400.708968:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187827200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203400.708973:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203400.708975:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203400.708977:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187827200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203401.412493:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203402.436526:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203403.460505:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203404.484450:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203405.508491:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203405.828433:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203405.828444:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203405.828448:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203405.828456:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4860 00000100:00080000:1.0:1788203405.828465:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203405.828471:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203405.828504:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187827328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203405.828530:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203405.828533:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203405.828535:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203405.828537:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4860 00000100:00080000:1.0:1788203405.828547:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203405.828551:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203405.828558:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203405.828569:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187827456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203405.828574:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187827328:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788203405.828582:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203405.828585:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203405.828586:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203405.828589:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4860 00000100:00080000:1.0:1788203405.828591:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203405.828593:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203405.828596:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203405.828603:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187827584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203405.828609:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203405.828612:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203405.828613:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203405.828617:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4860 00000100:00080000:1.0:1788203405.828631:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203405.828634:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203405.828638:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788203405.828643:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203405.828648:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187827712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203405.828656:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203405.828658:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203405.828660:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203405.828663:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4860 00000100:00080000:1.0:1788203405.828665:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203405.828668:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203405.828670:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203405.828677:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187827840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203405.828690:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203405/real 1788203405] req@ffff8e828d720e00 x1875062187827328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203405.828709:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187827328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203405.828720:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203405.828724:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203405.828729:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187827328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203405.828752:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187827456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203405.828791:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203405.828804:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203405/real 1788203405] req@ffff8e828d723800 x1875062187827456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203405.828818:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187827456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203405.828826:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203405.828830:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203405.828834:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187827456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203405.828840:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187827584:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203405.828875:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203405.828886:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203405/real 1788203405] req@ffff8e828d721500 x1875062187827584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203405.828897:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187827584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203405.828906:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203405.828909:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203405.828913:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187827584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203405.828919:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187827712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203405.828938:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203405.828949:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203405/real 1788203405] req@ffff8e828d720000 x1875062187827712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203405.828959:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187827712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203405.828966:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203405.828969:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203405.828972:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187827712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203405.828977:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187827840:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203405.828992:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203405.829000:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203405/real 1788203405] req@ffff8e828d722680 x1875062187827840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203405.829012:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187827840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203405.829020:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203405.829023:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203405.829026:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187827840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203406.532444:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203407.556495:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203408.580483:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203409.604497:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203410.628527:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203410.948441:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203410.948450:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203410.948454:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203410.948462:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4866 00000100:00080000:1.0:1788203410.948472:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203410.948478:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203410.948512:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187827968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203410.948539:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203410.948542:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203410.948543:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203410.948547:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4866 00000100:00080000:1.0:1788203410.948549:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203410.948552:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203410.948559:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203410.948568:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187828096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203410.948575:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203410.948578:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203410.948579:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203410.948581:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4866 00000100:00080000:1.0:1788203410.948583:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203410.948585:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203410.948588:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203410.948594:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187828224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203410.948601:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187827968:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788203410.948603:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203410.948609:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203410.948613:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203410.948616:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4866 00000100:00080000:1.0:1788203410.948618:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203410.948620:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203410.948623:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203410.948630:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187828352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203410.948636:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203410.948638:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203410.948639:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203410.948641:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4866 00000100:00080000:1.0:1788203410.948643:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203410.948645:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203410.948648:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203410.948654:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187828480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203410.948705:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203410.948758:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203410/real 1788203410] req@ffff8e828d722680 x1875062187827968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203410.948774:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187827968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203410.948782:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203410.948785:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203410.948788:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187827968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203410.948794:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187828096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203410.948812:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203410.948821:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203410/real 1788203410] req@ffff8e828d720000 x1875062187828096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203410.948829:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187828096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203410.948835:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203410.948836:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203410.948839:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187828096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203410.948860:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187828224:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203410.948879:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203410.948890:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203410/real 1788203410] req@ffff8e828d721500 x1875062187828224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203410.948903:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187828224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203410.948912:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203410.948916:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203410.948919:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187828224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203410.948924:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187828352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203410.948939:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203410.948946:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203410/real 1788203410] req@ffff8e828d723800 x1875062187828352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203410.948954:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187828352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203410.948959:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203410.948961:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203410.948964:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187828352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203410.948967:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187828480:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203410.948976:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203410.948982:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203410/real 1788203410] req@ffff8e828d720e00 x1875062187828480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203410.948989:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187828480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203410.948994:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203410.948995:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203410.948997:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187828480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203411.652596:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203412.676561:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203413.700502:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203414.724479:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203415.748512:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203416.068426:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203416.068434:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203416.068438:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203416.068446:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4871 00000100:00080000:2.0:1788203416.068455:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203416.068460:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203416.068491:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187828608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203416.068512:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203416.068516:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203416.068518:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203416.068523:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4871 00000100:00080000:2.0:1788203416.068526:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203416.068530:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203416.068539:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203416.068552:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187828736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203416.068560:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788203416.068562:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187828608:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203416.068563:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203416.068565:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203416.068569:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4871 00000100:00080000:2.0:1788203416.068572:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203416.068575:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203416.068579:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203416.068619:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187828864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203416.068629:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203416.068632:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203416.068634:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203416.068638:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4871 00000800:00000100:0.0:1788203416.068639:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203416.068643:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203416.068648:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203416.068653:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203416.068662:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187828992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203416.068670:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203416.068673:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203416.068674:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203416.068679:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4871 00000100:00080000:2.0:1788203416.068682:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203416.068685:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203416.068687:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203416/real 1788203416] req@ffff8e82894e4000 x1875062187828608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788203416.068690:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203416.068699:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187829120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203416.068707:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187828608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203416.068717:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203416.068722:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203416.068726:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187828608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203416.068748:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187828736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203416.068766:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203416.068776:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203416/real 1788203416] req@ffff8e82894e4e00 x1875062187828736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203416.068783:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187828736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203416.068789:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203416.068791:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203416.068794:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187828736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203416.068797:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187828864:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203416.068808:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203416.068814:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203416/real 1788203416] req@ffff8e82894e4a80 x1875062187828864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203416.068826:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187828864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203416.068831:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203416.068833:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203416.068834:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187828864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203416.068838:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187828992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203416.068848:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203416.068854:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203416/real 1788203416] req@ffff8e82894e7800 x1875062187828992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203416.068861:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187828992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203416.068866:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203416.068868:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203416.068869:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187828992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203416.068872:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187829120:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203416.068881:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203416.068887:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203416/real 1788203416] req@ffff8e82894e6300 x1875062187829120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203416.068894:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187829120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203416.068903:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203416.068905:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203416.068907:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187829120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203416.772494:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203417.796516:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203418.820493:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203419.844481:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203420.868544:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203421.188410:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203421.188419:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203421.188422:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203421.188431:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4876 00000100:00080000:0.0:1788203421.188440:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203421.188445:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203421.188476:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187829248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203421.188511:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203421.188515:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203421.188516:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203421.188519:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4876 00000100:00080000:0.0:1788203421.188522:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203421.188525:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203421.188533:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203421.188545:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187829376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203421.188553:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203421.188556:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203421.188557:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203421.188564:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4876 00000100:00080000:0.0:1788203421.188568:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788203421.188569:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187829248:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203421.188573:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203421.188580:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203421.188588:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187829504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203421.188595:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203421.188597:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203421.188599:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203421.188601:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4876 00000100:00080000:0.0:1788203421.188603:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203421.188605:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203421.188608:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203421.188615:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187829632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203421.188621:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203421.188623:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203421.188624:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203421.188627:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4876 00000100:00080000:0.0:1788203421.188632:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203421.188634:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203421.188638:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788203421.188645:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203421.188648:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187829760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203421.188707:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203421/real 1788203421] req@ffff8e828793f800 x1875062187829248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203421.188724:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187829248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203421.188735:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203421.188739:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203421.188744:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187829248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203421.188752:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187829376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203421.188776:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203421.188791:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203421/real 1788203421] req@ffff8e828793ce00 x1875062187829376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203421.188803:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187829376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203421.188812:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203421.188815:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203421.188819:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187829376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203421.188841:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187829504:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203421.188861:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203421.188871:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203421/real 1788203421] req@ffff8e828793c700 x1875062187829504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203421.188883:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187829504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203421.188891:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203421.188895:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203421.188898:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187829504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203421.188904:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187829632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203421.188921:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203421.188930:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203421/real 1788203421] req@ffff8e828793f100 x1875062187829632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203421.188941:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187829632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203421.188949:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203421.188952:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203421.188956:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187829632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203421.188961:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187829760:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203421.188976:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203421.188986:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203421/real 1788203421] req@ffff8e828793e680 x1875062187829760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203421.188996:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187829760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203421.189004:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203421.189007:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203421.189010:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187829760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203421.892471:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203422.916556:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203423.940469:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203424.964530:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203425.988544:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203426.308401:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203426.308422:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203426.308426:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203426.308435:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4881 00000100:00080000:0.0:1788203426.308444:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203426.308449:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203426.308482:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187829888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203426.308498:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203426.308501:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203426.308502:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203426.308505:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4881 00000100:00080000:0.0:1788203426.308508:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203426.308511:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203426.308519:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203426.308529:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187830016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203426.308532:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187829888:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203426.308544:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203426.308547:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203426.308548:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203426.308552:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4881 00000100:00080000:0.0:1788203426.308555:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203426.308557:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203426.308560:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203426.308570:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187830144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203426.308576:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203426.308579:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203426.308581:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203426.308584:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4881 00000100:00080000:0.0:1788203426.308586:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203426.308588:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203426.308592:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788203426.308594:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203426.308603:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187830272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203426.308611:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203426.308613:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203426.308614:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203426.308616:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4881 00000100:00080000:0.0:1788203426.308619:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203426.308621:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203426.308625:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203426.308631:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187830400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203426.308642:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203426/real 1788203426] req@ffff8e828793ca80 x1875062187829888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203426.308663:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187829888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203426.308674:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203426.308679:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203426.308684:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187829888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203426.308705:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187830016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203426.308723:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203426.308733:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203426/real 1788203426] req@ffff8e828793c000 x1875062187830016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203426.308741:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187830016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203426.308747:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203426.308750:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203426.308752:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187830016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203426.308755:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187830144:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203426.308766:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203426.308773:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203426/real 1788203426] req@ffff8e828793dc00 x1875062187830144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203426.308780:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187830144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203426.308785:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203426.308787:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203426.308789:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187830144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203426.308792:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187830272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203426.308802:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203426.308807:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203426/real 1788203426] req@ffff8e828793f480 x1875062187830272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203426.308815:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187830272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203426.308820:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203426.308822:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203426.308824:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187830272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203426.308827:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187830400:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203426.308836:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203426.308841:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203426/real 1788203426] req@ffff8e828793c380 x1875062187830400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203426.308848:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187830400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203426.308853:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203426.308855:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203426.308857:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187830400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203427.012574:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203428.036580:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203429.060535:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203430.084472:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203431.108568:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203431.428435:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203431.428453:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203431.428458:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203431.428469:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4886 00000100:00080000:2.0:1788203431.428483:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203431.428493:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203431.428541:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187830528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203431.428576:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203431.428582:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203431.428585:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203431.428589:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4886 00000100:00080000:2.0:1788203431.428593:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203431.428597:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203431.428609:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203431.428623:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187830656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203431.428633:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203431.428636:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203431.428637:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203431.428640:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4886 00000100:00080000:2.0:1788203431.428643:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203431.428648:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203431.428658:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203431.428667:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187830528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203431.428668:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187830784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203431.428688:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203431.428691:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203431.428692:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203431.428696:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4886 00000100:00080000:2.0:1788203431.428699:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203431.428703:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203431.428708:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203431.428722:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187830912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203431.428729:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203431.428732:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203431.428734:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203431.428741:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4886 00000100:00080000:2.0:1788203431.428745:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203431.428749:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203431.428752:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203431.428765:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187831040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203431.428768:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203431.428833:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203431/real 1788203431] req@ffff8e82894e7100 x1875062187830528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203431.428848:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187830528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203431.428857:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203431.428861:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203431.428864:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187830528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203431.428870:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187830656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203431.428886:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203431.428897:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203431/real 1788203431] req@ffff8e82894e5c00 x1875062187830656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203431.428906:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187830656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203431.428912:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203431.428914:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203431.428916:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187830656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203431.428941:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187830784:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203431.428958:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203431.428969:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203431/real 1788203431] req@ffff8e82894e4380 x1875062187830784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203431.428980:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187830784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203431.428985:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203431.428987:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203431.428990:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187830784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203431.428996:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187830912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203431.429009:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203431.429016:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203431/real 1788203431] req@ffff8e82894e5f80 x1875062187830912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203431.429026:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187830912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203431.429033:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203431.429035:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203431.429037:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187830912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203431.429041:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187831040:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203431.429053:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203431.429063:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203431/real 1788203431] req@ffff8e82894e6d80 x1875062187831040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203431.429078:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187831040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203431.429085:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203431.429088:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203431.429091:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187831040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203432.132469:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203433.156483:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203434.180482:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203435.204525:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203436.228439:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203436.548454:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203436.548463:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203436.548469:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203436.548478:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4891 00000100:00080000:0.0:1788203436.548488:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203436.548494:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203436.548528:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187831168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203436.548551:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203436.548554:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203436.548555:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203436.548558:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4891 00000100:00080000:0.0:1788203436.548560:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203436.548563:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203436.548570:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203436.548578:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187831296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203436.548585:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203436.548588:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203436.548592:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203436.548594:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4891 00000100:00080000:0.0:1788203436.548598:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203436.548601:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203436.548604:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203436.548615:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187831424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203436.548621:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187831168:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203436.548625:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203436.548640:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203436.548641:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203436.548645:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4891 00000100:00080000:0.0:1788203436.548647:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203436.548649:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203436.548653:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203436.548666:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187831552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203436.548676:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203436.548680:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203436.548682:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203436.548688:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4891 00000100:00080000:0.0:1788203436.548707:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203436.548713:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203436.548720:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203436.548735:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187831680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203436.548760:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187831296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203436.548805:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187831424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203436.548834:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187831552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203436.548862:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187831680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203437.252496:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203438.276487:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203439.300509:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788203439.620778:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:1.0:1788203439.620846:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788203439.620871:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788203439.620926:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788203439.620938:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788203439.620947:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788203439.620958:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:3.0:1788203439.621041:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203436/real 1788203439] req@ffff8e828793fb80 x1875062187831424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203439.621080:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187831424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203439.621093:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203439.621098:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203439.621103:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187831424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203439.621111:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203436/real 1788203439] req@ffff8e828793d880 x1875062187831552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203439.621123:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187831552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203439.621131:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203439.621135:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203439.621151:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187831552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203439.621159:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203436/real 1788203439] req@ffff8e828793c380 x1875062187831680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203439.621173:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187831680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203439.621195:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203439.621198:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203439.621201:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187831680:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203439.621207:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203436/real 1788203439] req@ffff8e828793e300 x1875062187831168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203439.621219:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187831168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203439.621226:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203439.621229:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203439.621234:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187831168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203439.621240:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203436/real 1788203439] req@ffff8e828793d500 x1875062187831296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203439.621251:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187831296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203439.621258:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203439.621261:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203439.621263:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187831296:192.168.201.121@tcp:38:kworker.0 00010000:00010000:0.0:1788203440.324486:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203441.348495:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203441.668462:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203441.668479:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203441.668482:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203441.668490:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4896 00000100:00080000:1.0:1788203441.668499:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203441.668504:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203441.668531:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187831808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203441.668555:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203441.668558:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203441.668559:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203441.668563:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4896 00000100:00080000:1.0:1788203441.668567:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203441.668571:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203441.668580:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203441.668595:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187831936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203441.668605:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203441.668609:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203441.668612:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788203441.668616:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187831808:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788203441.668618:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4896 00000100:00080000:1.0:1788203441.668624:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203441.668632:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203441.668637:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203441.668651:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187832064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203441.668661:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203441.668665:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203441.668666:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203441.668671:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4896 00000100:00080000:1.0:1788203441.668675:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203441.668679:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203441.668685:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203441.668698:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187832192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203441.668699:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788203441.668712:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203441.668715:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203441.668718:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203441.668722:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4896 00000100:00080000:1.0:1788203441.668726:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203441.668730:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203441.668734:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203441.668747:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187832320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203441.668752:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203441/real 1788203441] req@ffff8e828d720380 x1875062187831808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203441.668778:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187831808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203441.668792:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203441.668797:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203441.668801:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187831808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203441.668827:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187831936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203441.668853:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203441.668868:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203441/real 1788203441] req@ffff8e828d722300 x1875062187831936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203441.668890:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187831936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203441.668899:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203441.668902:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203441.668906:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187831936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203441.668912:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187832064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203441.668934:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203441.668945:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203441/real 1788203441] req@ffff8e828d722d80 x1875062187832064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203441.668956:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187832064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203441.668963:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203441.668966:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203441.668969:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187832064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203441.668975:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187832192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203441.668993:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203441.669003:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203441/real 1788203441] req@ffff8e828d721180 x1875062187832192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203441.669013:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187832192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203441.669027:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203441.669030:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203441.669033:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187832192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203441.669038:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187832320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203441.669055:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203441.669066:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203441/real 1788203441] req@ffff8e828d720a80 x1875062187832320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203441.669078:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187832320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203441.669086:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203441.669089:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203441.669093:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187832320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203442.372428:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203443.396455:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203444.420562:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203445.444451:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203446.468448:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203446.788388:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203446.788396:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203446.788400:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203446.788408:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4901 00000100:00080000:1.0:1788203446.788417:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203446.788423:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203446.788454:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187832448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203446.788476:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203446.788478:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203446.788480:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203446.788483:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4901 00000100:00080000:1.0:1788203446.788485:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203446.788488:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203446.788496:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203446.788508:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187832576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203446.788510:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187832448:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788203446.788523:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203446.788526:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203446.788527:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203446.788529:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4901 00000100:00080000:1.0:1788203446.788532:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203446.788534:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203446.788545:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203446.788558:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187832704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203446.788565:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203446.788568:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203446.788569:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788203446.788572:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788203446.788574:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4901 00000100:00080000:1.0:1788203446.788578:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203446.788581:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203446.788586:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203446.788594:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187832832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203446.788600:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203446.788603:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203446.788604:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203446.788607:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4901 00000100:00080000:1.0:1788203446.788610:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788203446.788612:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203446/real 1788203446] req@ffff8e828d720a80 x1875062187832448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203446.788615:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203446.788622:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203446.788635:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187832960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203446.788646:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187832448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203446.788656:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203446.788660:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203446.788663:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187832448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203446.788683:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187832576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203446.788698:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203446.788707:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203446/real 1788203446] req@ffff8e828d721180 x1875062187832576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203446.788715:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187832576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203446.788720:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203446.788722:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203446.788725:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187832576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203446.788728:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187832704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203446.788738:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203446.788744:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203446/real 1788203446] req@ffff8e828d722d80 x1875062187832704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203446.788751:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187832704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203446.788756:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203446.788758:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203446.788760:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187832704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203446.788763:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187832832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203446.788774:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203446.788779:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203446/real 1788203446] req@ffff8e828d722300 x1875062187832832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203446.788787:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187832832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203446.788791:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203446.788793:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203446.788795:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187832832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203446.788798:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187832960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203446.788808:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203446.788813:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203446/real 1788203446] req@ffff8e828d720380 x1875062187832960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203446.788819:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187832960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203446.788825:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203446.788827:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203446.788831:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187832960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203447.492433:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203448.516484:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203449.540471:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203450.564497:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203451.588444:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203451.908443:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203451.908453:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203451.908457:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203451.908469:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4906 00000100:00080000:1.0:1788203451.908479:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203451.908487:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203451.908525:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187833088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203451.908548:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203451.908552:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203451.908553:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203451.908557:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4906 00000100:00080000:1.0:1788203451.908560:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203451.908563:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203451.908570:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203451.908579:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187833216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203451.908587:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203451.908590:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203451.908591:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203451.908594:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4906 00000100:00080000:1.0:1788203451.908597:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203451.908600:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203451.908604:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203451.908612:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187833344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203451.908615:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187833088:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788203451.908632:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203451.908634:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203451.908635:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203451.908638:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4906 00000100:00080000:1.0:1788203451.908639:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203451.908642:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203451.908646:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203451.908655:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187833472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203451.908661:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203451.908664:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203451.908665:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203451.908667:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4906 00000100:00080000:1.0:1788203451.908669:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203451.908672:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203451.908675:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203451.908682:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187833600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203451.908686:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203451.908733:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203451/real 1788203451] req@ffff8e828d720e00 x1875062187833088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203451.908748:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187833088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203451.908761:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203451.908764:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203451.908767:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187833088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203451.908772:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187833216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203451.908787:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203451.908795:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203451/real 1788203451] req@ffff8e828d723800 x1875062187833216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203451.908803:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187833216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203451.908809:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203451.908811:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203451.908813:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187833216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203451.908832:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187833344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203451.908842:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203451.908849:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203451/real 1788203451] req@ffff8e828d721500 x1875062187833344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203451.908856:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187833344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203451.908862:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203451.908864:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203451.908866:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187833344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203451.908870:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187833472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203451.908889:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203451.908900:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203451/real 1788203451] req@ffff8e828d720000 x1875062187833472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203451.908913:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187833472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203451.908922:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203451.908925:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203451.908929:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187833472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203451.908934:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187833600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203451.908950:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203451.908958:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203451/real 1788203451] req@ffff8e828d722680 x1875062187833600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203451.908980:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187833600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203451.908986:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203451.908988:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203451.908991:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187833600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203452.612492:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203453.636421:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203454.660478:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203455.684518:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203456.708545:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203457.028472:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203457.028487:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203457.028493:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203457.028508:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4912 00000100:00080000:1.0:1788203457.028523:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203457.028533:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203457.028586:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187833728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203457.028621:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203457.028627:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203457.028629:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203457.028634:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4912 00000100:00080000:1.0:1788203457.028640:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203457.028646:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203457.028657:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203457.028673:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187833856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203457.028673:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187833728:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788203457.028707:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203457.028710:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203457.028712:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203457.028716:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4912 00000100:00080000:1.0:1788203457.028721:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203457.028724:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203457.028727:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203457.028738:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187833984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203457.028749:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203457.028752:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203457.028754:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203457.028757:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4912 00000100:00080000:1.0:1788203457.028759:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203457.028762:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203457.028765:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203457.028771:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187834112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203457.028782:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788203457.028786:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203457.028789:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203457.028792:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203457.028795:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4912 00000100:00080000:1.0:1788203457.028799:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203457.028803:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203457.028808:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203457.028820:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187834240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203457.028843:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203457/real 1788203457] req@ffff8e828d722680 x1875062187833728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203457.028866:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187833728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203457.028889:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203457.028894:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203457.028905:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187833728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203457.028933:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187833856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203457.028956:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203457.028968:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203457/real 1788203457] req@ffff8e828d720000 x1875062187833856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203457.028979:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187833856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203457.028987:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203457.028990:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203457.028993:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187833856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203457.028999:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187833984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203457.029013:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203457.029021:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203457/real 1788203457] req@ffff8e828d721500 x1875062187833984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203457.029031:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187833984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203457.029045:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203457.029048:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203457.029051:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187833984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203457.029056:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187834112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203457.029070:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203457.029078:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203457/real 1788203457] req@ffff8e828d723800 x1875062187834112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203457.029089:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187834112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203457.029097:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203457.029100:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203457.029103:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187834112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203457.029109:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187834240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203457.029127:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203457.029137:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203457/real 1788203457] req@ffff8e828d720e00 x1875062187834240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203482 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203457.029148:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187834240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203482 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203457.029155:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203457.029158:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203457.029162:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187834240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203457.732560:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203458.756463:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203459.780498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203460.804462:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203461.828451:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203462.148445:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203462.148456:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203462.148461:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203462.148472:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4917 00000100:00080000:0.0:1788203462.148483:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203462.148491:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203462.148530:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187834368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203462.148559:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203462.148564:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203462.148567:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203462.148572:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4917 00000100:00080000:0.0:1788203462.148576:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203462.148581:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203462.148590:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203462.148606:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187834496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203462.148618:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203462.148622:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203462.148624:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203462.148629:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4917 00000100:00080000:0.0:1788203462.148633:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203462.148638:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203462.148644:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203462.148657:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187834368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203462.148661:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187834624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203462.148681:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203462.148685:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203462.148687:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203462.148693:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4917 00000100:00080000:0.0:1788203462.148697:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203462.148702:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203462.148708:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203462.148723:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187834752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203462.148734:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203462.148738:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203462.148740:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203462.148745:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4917 00000100:00080000:0.0:1788203462.148750:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788203462.148753:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203462.148760:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203462.148769:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203462.148805:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187834880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203462.148840:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203462/real 1788203462] req@ffff8e828793f480 x1875062187834368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203462.148862:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187834368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203462.148875:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203462.148879:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203462.148882:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187834368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203462.148890:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187834496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203462.148918:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203462.148933:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203462/real 1788203462] req@ffff8e828793dc00 x1875062187834496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203462.148947:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187834496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203462.148957:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203462.148960:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203462.148963:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187834496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203462.148990:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187834624:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203462.149008:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203462.149017:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203462/real 1788203462] req@ffff8e828793c000 x1875062187834624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203462.149026:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187834624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203462.149032:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203462.149034:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203462.149036:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187834624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203462.149040:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187834752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203462.149052:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203462.149059:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203462/real 1788203462] req@ffff8e828793ca80 x1875062187834752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203462.149066:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187834752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203462.149071:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203462.149073:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203462.149075:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187834752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203462.149079:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187834880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203462.149089:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203462.149095:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203462/real 1788203462] req@ffff8e828793e680 x1875062187834880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203462.149102:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187834880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203462.149108:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203462.149110:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203462.149112:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187834880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203462.852463:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203463.876501:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203464.900491:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203465.924496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203466.948488:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203467.268410:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203467.268418:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203467.268422:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203467.268431:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4922 00000100:00080000:0.0:1788203467.268442:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203467.268448:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203467.268479:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187835008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203467.268499:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203467.268502:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203467.268504:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203467.268508:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4922 00000100:00080000:0.0:1788203467.268510:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203467.268513:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203467.268520:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203467.268532:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187835136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203467.268543:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203467.268546:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203467.268548:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203467.268553:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4922 00000100:00080000:0.0:1788203467.268558:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203467.268562:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203467.268569:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203467.268584:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187835264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203467.268594:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203467.268598:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203467.268600:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203467.268604:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4922 00000100:00080000:0.0:1788203467.268609:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203467.268623:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203467.268632:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203467.268646:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187835392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203467.268656:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203467.268660:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203467.268662:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203467.268667:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4922 00000100:00080000:0.0:1788203467.268671:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203467.268676:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203467.268682:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203467.268695:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187835520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203467.268942:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187835008:192.168.201.121@tcp:250:kworker.0 00000800:00000100:3.0:1788203467.269043:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203467.269091:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203467/real 1788203467] req@ffff8e828793f100 x1875062187835008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203467.269110:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187835008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203467.269117:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203467.269120:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203467.269123:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187835008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203467.269128:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187835136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203467.269146:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203467.269155:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203467/real 1788203467] req@ffff8e828793c700 x1875062187835136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203467.269163:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187835136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203467.269168:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203467.269170:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203467.269172:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187835136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203467.269176:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187835264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203467.269187:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203467.269193:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203467/real 1788203467] req@ffff8e828793ce00 x1875062187835264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203467.269200:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187835264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203467.269205:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203467.269207:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203467.269209:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187835264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203467.269212:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187835392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203467.269222:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203467.269227:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203467/real 1788203467] req@ffff8e828793f800 x1875062187835392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203467.269234:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187835392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203467.269239:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203467.269241:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203467.269242:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187835392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203467.269246:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187835520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203467.269255:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203467.269260:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203467/real 1788203467] req@ffff8e828793e680 x1875062187835520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203467.269267:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187835520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203467.269272:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203467.269274:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203467.269275:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187835520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203467.972561:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203468.996474:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203470.020480:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203471.044430:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203472.068484:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203472.388438:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203472.388449:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203472.388452:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203472.388463:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4927 00000100:00080000:2.0:1788203472.388474:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203472.388482:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203472.388519:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187835648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203472.388545:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203472.388549:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203472.388551:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203472.388555:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4927 00000100:00080000:2.0:1788203472.388570:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203472.388580:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203472.388588:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187835648:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788203472.388589:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203472.388625:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187835776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203472.388638:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203472.388642:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203472.388644:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203472.388648:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4927 00000100:00080000:2.0:1788203472.388664:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203472.388669:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203472.388674:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203472.388680:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203472.388690:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187835904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203472.388702:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203472.388705:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203472.388708:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203472.388712:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4927 00000100:00080000:2.0:1788203472.388716:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203472.388721:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203472.388726:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203472.388729:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203472/real 1788203472] req@ffff8e82894e6d80 x1875062187835648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203472.388740:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187836032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203472.388756:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187835648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203472.388762:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203472.388772:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203472.388776:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203472.388781:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203472.388783:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4927 00000100:00080000:2.0:1788203472.388789:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203472.388791:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:2.0:1788203472.388797:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203472.388802:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187835648:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788203472.388806:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203472.388821:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187836160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203472.388832:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187835776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203472.388860:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203472.388875:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203472/real 1788203472] req@ffff8e82894e5f80 x1875062187835776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203472.388888:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187835776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203472.388905:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203472.388909:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203472.388913:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187835776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203472.388920:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187835904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203472.388940:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203472.388952:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203472/real 1788203472] req@ffff8e82894e4380 x1875062187835904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203472.388965:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187835904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203472.388975:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203472.388978:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203472.388982:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187835904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203472.388988:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187836032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203472.389009:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203472.389019:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203472/real 1788203472] req@ffff8e82894e5c00 x1875062187836032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203472.389032:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187836032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203472.389041:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203472.389044:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203472.389049:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187836032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203472.389065:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187836160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203472.389085:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203472.389097:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203472/real 1788203472] req@ffff8e82894e7100 x1875062187836160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203472.389110:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187836160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203472.389119:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203472.389123:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203472.389126:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187836160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203473.092481:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203474.116490:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203475.140520:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203476.164531:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203477.188444:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203477.508396:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203477.508406:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203477.508410:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203477.508418:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4932 00000100:00080000:2.0:1788203477.508427:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203477.508434:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203477.508465:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187836288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203477.508489:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203477.508492:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203477.508494:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203477.508497:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4932 00000100:00080000:2.0:1788203477.508499:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203477.508501:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203477.508510:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203477.508518:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187836416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203477.508539:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203477.508541:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203477.508543:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203477.508545:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4932 00000100:00080000:2.0:1788203477.508547:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203477.508550:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203477.508553:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203477.508561:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187836544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203477.508567:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203477.508569:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203477.508570:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203477.508572:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4932 00000100:00080000:2.0:1788203477.508574:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203477.508576:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203477.508578:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203477.508583:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187836672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203477.508588:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203477.508590:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203477.508592:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203477.508595:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4932 00000100:00080000:2.0:1788203477.508598:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203477.508600:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203477.508603:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203477.508611:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187836288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203477.508612:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187836800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203477.508670:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203477.508710:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203477/real 1788203477] req@ffff8e82894e6300 x1875062187836288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203477.508723:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187836288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203477.508731:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203477.508734:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203477.508738:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187836288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203477.508743:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187836416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203477.508757:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203477.508766:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203477/real 1788203477] req@ffff8e82894e7800 x1875062187836416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203477.508773:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187836416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203477.508779:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203477.508780:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203477.508782:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187836416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203477.508786:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187836544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203477.508797:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203477.508803:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203477/real 1788203477] req@ffff8e82894e4a80 x1875062187836544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203477.508810:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187836544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203477.508815:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203477.508817:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203477.508819:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187836544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203477.508822:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187836672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203477.508833:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203477.508838:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203477/real 1788203477] req@ffff8e82894e4e00 x1875062187836672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203477.508850:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187836672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203477.508859:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203477.508862:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203477.508866:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187836672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203477.508891:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187836800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203477.508910:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203477.508918:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203477/real 1788203477] req@ffff8e82894e4000 x1875062187836800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203477.508926:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187836800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203477.508931:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203477.508934:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203477.508936:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187836800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203478.212430:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203479.236483:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203480.260462:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203481.284494:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203482.308524:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203482.628472:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203482.628482:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203482.628486:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203482.628495:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4937 00000100:00080000:0.0:1788203482.628504:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203482.628511:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203482.628550:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187836928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203482.628586:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203482.628591:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203482.628593:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203482.628598:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4937 00000100:00080000:0.0:1788203482.628602:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203482.628607:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203482.628618:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203482.628628:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187836928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203482.628633:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187837056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203482.628643:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203482.628646:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203482.628647:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203482.628650:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4937 00000100:00080000:0.0:1788203482.628654:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203482.628656:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203482.628660:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203482.628668:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187837184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203482.628675:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203482.628678:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203482.628679:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203482.628681:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4937 00000100:00080000:0.0:1788203482.628684:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203482.628687:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203482.628691:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788203482.628699:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203482.628700:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187837312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203482.628709:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203482.628711:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203482.628713:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203482.628715:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4937 00000100:00080000:0.0:1788203482.628718:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203482.628720:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203482.628723:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203482.628730:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187837440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203482.628741:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203482/real 1788203482] req@ffff8e828793ca80 x1875062187836928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203482.628760:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187836928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203482.628771:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203482.628775:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203482.628779:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187836928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203482.628804:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187837056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203482.628830:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203482.628843:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203482/real 1788203482] req@ffff8e828793c000 x1875062187837056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203482.628857:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187837056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203482.628867:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203482.628870:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203482.628874:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187837056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203482.628881:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187837184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203482.628901:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203482.628912:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203482/real 1788203482] req@ffff8e828793dc00 x1875062187837184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203482.628924:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187837184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203482.628933:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203482.628936:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203482.628940:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187837184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203482.628947:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187837312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203482.628968:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203482.628979:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203482/real 1788203482] req@ffff8e828793f480 x1875062187837312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203482.628992:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187837312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203482.629002:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203482.629005:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203482.629009:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187837312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203482.629015:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187837440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203482.629035:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203482.629046:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203482/real 1788203482] req@ffff8e828793d500 x1875062187837440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203482.629065:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187837440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203482.629074:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203482.629077:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203482.629081:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187837440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203483.332479:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203484.356508:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203485.380523:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203486.404499:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203487.428507:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203487.748423:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203487.748433:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203487.748436:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203487.748446:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4942 00000100:00080000:2.0:1788203487.748455:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203487.748460:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203487.748503:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187837568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203487.748525:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203487.748528:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203487.748529:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203487.748532:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4942 00000100:00080000:2.0:1788203487.748535:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203487.748538:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203487.748545:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203487.748554:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187837696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203487.748566:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203487.748569:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203487.748570:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203487.748574:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4942 00000100:00100000:3.0:1788203487.748576:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187837568:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203487.748577:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203487.748581:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203487.748587:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203487.748594:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187837824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203487.748601:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203487.748604:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203487.748605:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203487.748607:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4942 00000100:00080000:2.0:1788203487.748609:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203487.748611:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203487.748614:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203487.748621:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187837952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203487.748627:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203487.748629:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203487.748630:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203487.748632:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4942 00000100:00080000:2.0:1788203487.748635:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203487.748638:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:3.0:1788203487.748642:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:2.0:1788203487.748643:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203487.748654:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187838080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203487.748690:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203487/real 1788203487] req@ffff8e82894e7480 x1875062187837568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203487.748705:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187837568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203487.748718:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203487.748720:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203487.748724:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187837568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203487.748742:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187837696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203487.748766:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203487.748775:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203487/real 1788203487] req@ffff8e82894e6a00 x1875062187837696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203487.748783:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187837696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203487.748789:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203487.748791:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203487.748793:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187837696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203487.748797:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187837824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203487.748808:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203487.748814:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203487/real 1788203487] req@ffff8e82894e5880 x1875062187837824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203487.748821:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187837824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203487.748826:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203487.748828:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203487.748829:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187837824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203487.748834:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187837952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203487.748845:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203487.748850:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203487/real 1788203487] req@ffff8e82894e6680 x1875062187837952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203487.748858:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187837952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203487.748862:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203487.748864:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203487.748866:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187837952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203487.748869:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187838080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203487.748879:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203487.748884:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203487/real 1788203487] req@ffff8e82894e4000 x1875062187838080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203487.748892:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187838080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203487.748896:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203487.748898:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203487.748900:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187838080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203488.452491:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203489.476483:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203490.500456:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203491.524468:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203492.548440:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203492.868417:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203492.868425:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203492.868430:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203492.868444:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4947 00000100:00080000:0.0:1788203492.868453:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203492.868459:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203492.868493:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187838208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203492.868518:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203492.868521:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203492.868522:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203492.868525:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4947 00000100:00080000:0.0:1788203492.868528:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203492.868531:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203492.868539:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203492.868548:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187838336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203492.868557:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187838208:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203492.868558:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203492.868566:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203492.868568:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203492.868575:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4947 00000100:00080000:0.0:1788203492.868578:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203492.868581:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203492.868585:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203492.868594:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187838464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203492.868600:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203492.868603:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203492.868604:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203492.868606:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4947 00000100:00080000:0.0:1788203492.868609:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203492.868611:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203492.868615:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788203492.868619:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203492.868626:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187838592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203492.868638:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203492.868640:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203492.868641:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203492.868644:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4947 00000100:00080000:0.0:1788203492.868646:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203492.868649:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203492.868652:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203492.868663:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203492/real 1788203492] req@ffff8e828793e300 x1875062187838208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203492.868673:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187838720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203492.868683:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187838208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203492.868695:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203492.868698:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203492.868701:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187838208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203492.868719:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187838336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203492.868734:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203492.868742:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203492/real 1788203492] req@ffff8e828793c380 x1875062187838336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203492.868750:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187838336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203492.868756:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203492.868758:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203492.868760:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187838336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203492.868763:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187838464:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203492.868773:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203492.868779:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203492/real 1788203492] req@ffff8e828793d880 x1875062187838464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203492.868786:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187838464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203492.868795:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203492.868797:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203492.868799:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187838464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203492.868802:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187838592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203492.868812:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203492.868817:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203492/real 1788203492] req@ffff8e828793fb80 x1875062187838592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203492.868824:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187838592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203492.868829:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203492.868831:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203492.868833:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187838592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203492.868836:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187838720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203492.868845:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203492.868851:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203492/real 1788203492] req@ffff8e828793d500 x1875062187838720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203492.868858:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187838720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203492.868863:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203492.868864:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203492.868867:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187838720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203493.572557:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203494.596530:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203495.620571:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203496.644565:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203497.668584:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203497.988466:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203497.988474:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203497.988478:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203497.988486:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4953 00000100:00080000:0.0:1788203497.988496:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203497.988502:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203497.988536:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187838848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203497.988556:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203497.988558:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203497.988560:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203497.988563:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4953 00000100:00080000:0.0:1788203497.988565:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203497.988568:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203497.988575:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203497.988585:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187838976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203497.988592:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203497.988595:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203497.988596:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203497.988598:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4953 00000100:00080000:0.0:1788203497.988600:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203497.988605:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203497.988605:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187838848:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788203497.988613:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203497.988627:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187839104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203497.988634:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203497.988636:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203497.988638:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203497.988640:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4953 00000100:00080000:0.0:1788203497.988645:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203497.988648:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203497.988650:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203497.988658:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187839232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203497.988664:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788203497.988669:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203497.988670:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203497.988674:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203497.988682:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4953 00000100:00080000:0.0:1788203497.988687:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203497.988689:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203497.988693:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203497.988705:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187839360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203497.988728:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203497/real 1788203497] req@ffff8e828793f480 x1875062187838848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203497.988744:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187838848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203497.988752:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203497.988755:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203497.988762:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187838848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203497.988770:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187838976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203497.988786:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203497.988794:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203497/real 1788203497] req@ffff8e828793dc00 x1875062187838976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203497.988802:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187838976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203497.988807:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203497.988809:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203497.988812:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187838976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203497.988829:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187839104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203497.988840:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203497.988847:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203497/real 1788203497] req@ffff8e828793c000 x1875062187839104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203497.988854:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187839104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203497.988859:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203497.988869:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203497.988874:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187839104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203497.988878:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187839232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203497.988890:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203497.988896:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203497/real 1788203497] req@ffff8e828793ca80 x1875062187839232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203497.988904:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187839232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203497.988909:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203497.988911:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203497.988913:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187839232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203497.988916:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187839360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203497.988930:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203497.988940:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203497/real 1788203497] req@ffff8e828793e680 x1875062187839360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203497.988951:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187839360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203497.988961:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203497.988964:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203497.988968:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187839360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203498.692584:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203499.716496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203500.740552:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203501.764452:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203502.788517:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203503.108465:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203503.108472:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203503.108476:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203503.108486:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4958 00000100:00080000:0.0:1788203503.108495:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203503.108501:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203503.108530:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187839488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203503.108553:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203503.108556:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203503.108557:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203503.108560:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4958 00000100:00080000:0.0:1788203503.108563:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203503.108565:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203503.108573:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203503.108583:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187839616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203503.108596:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203503.108599:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203503.108600:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203503.108603:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4958 00000100:00080000:0.0:1788203503.108606:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203503.108608:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203503.108611:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203503.108618:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187839488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203503.108622:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187839744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203503.108637:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203503.108640:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203503.108643:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203503.108647:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4958 00000100:00080000:0.0:1788203503.108652:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203503.108667:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203503.108677:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203503.108692:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187839872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203503.108702:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203503.108705:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203503.108707:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203503.108710:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4958 00000100:00080000:0.0:1788203503.108743:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203503.108749:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203503.108755:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203503.108766:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187840000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203503.108803:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187839616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203503.108952:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187839744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203503.108984:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187839872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203503.109013:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187840000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203503.812470:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203504.836488:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203505.860472:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788203506.180573:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:2.0:1788203506.180614:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788203506.180631:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788203506.180674:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788203506.180684:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788203506.180692:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788203506.180701:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:3.0:1788203506.180734:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203503/real 1788203506] req@ffff8e828793c700 x1875062187839744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203506.180763:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187839744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203506.180775:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203506.180779:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203506.180784:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187839744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203506.180792:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203503/real 1788203506] req@ffff8e828793f100 x1875062187839872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203506.180806:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187839872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203506.180821:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203506.180826:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203506.180829:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187839872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203506.180836:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203503/real 1788203506] req@ffff8e828793e680 x1875062187840000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203506.180849:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187840000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203506.180856:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203506.180860:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203506.180876:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187840000:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203506.180887:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203503/real 1788203506] req@ffff8e828793f800 x1875062187839488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203506.180903:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187839488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203506.180912:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203506.180915:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203506.180919:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187839488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203506.180926:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203503/real 1788203506] req@ffff8e828793ce00 x1875062187839616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203506.180937:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187839616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203506.180946:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203506.180949:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203506.180951:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187839616:192.168.201.121@tcp:38:kworker.0 00010000:00010000:1.0:1788203506.884504:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203507.908476:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203508.228444:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203508.228456:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203508.228460:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203508.228470:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4963 00000100:00080000:0.0:1788203508.228480:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203508.228488:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203508.228517:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187840128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203508.228540:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203508.228543:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203508.228544:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203508.228548:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4963 00000100:00080000:0.0:1788203508.228550:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203508.228553:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203508.228560:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203508.228579:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187840256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203508.228588:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203508.228593:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203508.228596:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203508.228608:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4963 00000100:00100000:3.0:1788203508.228608:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187840128:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203508.228613:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203508.228624:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203508.228630:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203508.228647:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187840384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203508.228657:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203508.228660:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203508.228662:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203508.228666:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4963 00000100:00080000:0.0:1788203508.228671:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203508.228676:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203508.228681:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203508.228696:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187840512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203508.228707:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203508.228711:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203508.228715:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788203508.228717:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203508.228723:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4963 00000100:00080000:0.0:1788203508.228729:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203508.228733:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203508.228741:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203508.228757:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187840640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203508.228814:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203508/real 1788203508] req@ffff8e828793ca80 x1875062187840128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203508.228840:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187840128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203508.228855:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203508.228861:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203508.228866:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187840128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203508.228905:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187840256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203508.228935:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203508.228952:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203508/real 1788203508] req@ffff8e828793c000 x1875062187840256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203508.228967:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187840256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203508.228978:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203508.228982:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203508.229005:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187840256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203508.229015:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187840384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203508.229040:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203508.229055:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203508/real 1788203508] req@ffff8e828793dc00 x1875062187840384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203508.229070:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187840384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203508.229080:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203508.229084:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203508.229087:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187840384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203508.229095:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187840512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203508.229118:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203508.229130:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203508/real 1788203508] req@ffff8e828793f480 x1875062187840512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203508.229143:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187840512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203508.229159:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203508.229163:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203508.229168:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187840512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203508.229177:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187840640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203508.229201:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203508.229223:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203508/real 1788203508] req@ffff8e828793d500 x1875062187840640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203508.229362:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187840640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203533 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203508.229374:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203508.229378:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203508.229383:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187840640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203508.932445:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203509.956454:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203510.980425:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203512.004513:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203513.028470:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203513.348437:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203513.348448:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203513.348452:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203513.348463:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4968 00000100:00080000:0.0:1788203513.348474:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203513.348482:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203513.348519:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187840768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203513.348551:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203513.348556:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203513.348558:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203513.348562:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4968 00000100:00080000:0.0:1788203513.348566:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203513.348571:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203513.348580:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203513.348594:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187840896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203513.348608:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187840768:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203513.348609:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203513.348617:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203513.348620:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203513.348625:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4968 00000100:00080000:0.0:1788203513.348629:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203513.348634:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203513.348640:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203513.348654:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187841024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203513.348665:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203513.348670:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203513.348672:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203513.348676:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4968 00000100:00080000:0.0:1788203513.348680:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788203513.348682:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203513.348687:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203513.348694:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203513.348707:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187841152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203513.348718:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203513.348722:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203513.348724:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203513.348728:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4968 00000100:00080000:0.0:1788203513.348732:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203513.348738:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203513.348743:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203513/real 1788203513] req@ffff8e828793fb80 x1875062187840768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788203513.348744:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203513.348760:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187841280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203513.348770:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187840768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203513.348790:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203513.348798:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203513.348804:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187840768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203513.348829:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187840896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203513.348852:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203513.348864:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203513/real 1788203513] req@ffff8e828793d880 x1875062187840896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203513.348876:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187840896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203513.348890:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203513.348894:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203513.348913:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187840896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203513.348921:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187841024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203513.348941:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203513.348953:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203513/real 1788203513] req@ffff8e828793c380 x1875062187841024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203513.348965:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187841024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203513.348974:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203513.348977:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203513.348980:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187841024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203513.348986:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187841152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203513.349018:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203513.349029:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203513/real 1788203513] req@ffff8e828793e300 x1875062187841152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203513.349040:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187841152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203513.349048:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203513.349051:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203513.349055:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187841152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203513.349061:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187841280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203513.349077:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203513.349086:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203513/real 1788203513] req@ffff8e828793d500 x1875062187841280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203513.349097:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187841280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203513.349105:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203513.349108:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203513.349111:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187841280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203514.052445:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203515.076508:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203516.100481:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203517.124494:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203518.148498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203518.468391:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203518.468399:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203518.468403:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203518.468411:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4973 00000100:00080000:0.0:1788203518.468420:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203518.468426:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203518.468457:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187841408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203518.468478:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203518.468480:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203518.468482:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203518.468485:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4973 00000100:00080000:0.0:1788203518.468487:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203518.468490:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203518.468498:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203518.468506:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187841536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203518.468513:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788203518.468516:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187841408:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203518.468517:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203518.468520:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203518.468527:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4973 00000100:00080000:0.0:1788203518.468529:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203518.468531:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203518.468535:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203518.468545:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187841664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203518.468555:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203518.468558:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203518.468561:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203518.468564:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4973 00000100:00080000:0.0:1788203518.468569:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203518.468574:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203518.468580:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788203518.468585:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203518.468609:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187841792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203518.468618:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203518.468620:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788203518.468623:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203518/real 1788203518] req@ffff8e828793f480 x1875062187841408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203518.468624:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203518.468631:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4973 00000100:00080000:0.0:1788203518.468639:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203518.468644:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203518.468648:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187841408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788203518.468650:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788203518.468662:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788203518.468666:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187841920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203518.468668:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203518.468676:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187841408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203518.468699:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187841536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203518.468715:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203518.468724:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203518/real 1788203518] req@ffff8e828793dc00 x1875062187841536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203518.468733:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187841536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203518.468739:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203518.468741:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203518.468743:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187841536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203518.468747:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187841664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203518.468756:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203518.468763:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203518/real 1788203518] req@ffff8e828793c000 x1875062187841664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203518.468770:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187841664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203518.468775:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203518.468776:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203518.468778:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187841664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203518.468782:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187841792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203518.468792:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203518.468798:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203518/real 1788203518] req@ffff8e828793ca80 x1875062187841792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203518.468805:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187841792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203518.468836:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203518.468838:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203518.468840:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187841792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203518.468845:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187841920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203518.468859:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203518.468867:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203518/real 1788203518] req@ffff8e828793ce00 x1875062187841920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203518.468874:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187841920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203518.468879:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203518.468889:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203518.468891:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187841920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203519.172484:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203520.196447:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203521.220498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203522.244549:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203523.268497:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203523.588472:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203523.588480:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203523.588483:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203523.588492:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4978 00000100:00080000:2.0:1788203523.588501:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203523.588507:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203523.588541:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187842048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203523.588572:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203523.588575:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203523.588577:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203523.588580:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4978 00000100:00080000:2.0:1788203523.588583:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203523.588587:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203523.588595:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203523.588605:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187842176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203523.588613:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203523.588617:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203523.588619:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203523.588622:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4978 00000100:00100000:3.0:1788203523.588622:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187842048:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203523.588626:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203523.588635:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203523.588639:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203523.588647:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187842304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203523.588657:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203523.588660:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203523.588663:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203523.588667:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4978 00000100:00080000:2.0:1788203523.588671:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203523.588676:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203523.588682:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203523.588696:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187842432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203523.588713:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203523.588717:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203523.588730:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203523.588736:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4978 00000800:00000100:3.0:1788203523.588745:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203523.588752:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203523.588761:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203523.588767:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203523.588783:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187842560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203523.588816:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203523/real 1788203523] req@ffff8e82894e4e00 x1875062187842048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203523.588842:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187842048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203523.588855:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203523.588860:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203523.588865:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187842048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203523.588897:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187842176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203523.588922:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203523.588935:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203523/real 1788203523] req@ffff8e82894e4a80 x1875062187842176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203523.588946:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187842176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203523.588954:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203523.588957:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203523.588961:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187842176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203523.588968:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187842304:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203523.588988:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203523.588999:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203523/real 1788203523] req@ffff8e82894e7800 x1875062187842304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203523.589010:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187842304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203523.589017:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203523.589021:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203523.589024:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187842304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203523.589030:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187842432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203523.589052:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203523.589063:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203523/real 1788203523] req@ffff8e82894e6300 x1875062187842432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203523.589074:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187842432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203523.589081:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203523.589084:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203523.589087:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187842432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203523.589093:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187842560:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203523.589110:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203523.589119:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203523/real 1788203523] req@ffff8e82894e7100 x1875062187842560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203523.589129:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187842560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203523.589136:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203523.589139:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203523.589142:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187842560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203524.292534:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203525.316563:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203526.340486:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203527.364447:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203528.388487:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203528.708402:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203528.708413:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203528.708418:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203528.708436:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4983 00000100:00080000:0.0:1788203528.708447:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203528.708455:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203528.708492:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187842688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203528.708521:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203528.708526:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203528.708528:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203528.708532:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4983 00000100:00080000:0.0:1788203528.708536:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203528.708541:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203528.708551:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203528.708566:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187842816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203528.708577:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187842688:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203528.708580:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203528.708588:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203528.708590:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203528.708595:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4983 00000100:00080000:0.0:1788203528.708598:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203528.708603:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203528.708609:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203528.708622:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187842944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203528.708649:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203528.708651:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203528.708653:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203528.708655:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4983 00000100:00080000:0.0:1788203528.708658:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203528.708663:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:3.0:1788203528.708665:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788203528.708668:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203528.708680:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187843072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203528.708687:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203528.708690:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203528.708691:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203528.708693:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4983 00000100:00080000:0.0:1788203528.708696:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203528.708698:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203528.708701:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203528.708713:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187843200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203528.708714:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203528/real 1788203528] req@ffff8e828793f800 x1875062187842688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203528.708744:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187842688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203528.708773:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203528.708778:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203528.708782:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187842688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203528.708806:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187842816:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203528.708830:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203528.708844:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203528/real 1788203528] req@ffff8e828793e680 x1875062187842816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203528.708858:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187842816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203528.708867:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203528.708871:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203528.708875:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187842816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203528.708881:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187842944:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203528.708902:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203528.708914:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203528/real 1788203528] req@ffff8e828793f100 x1875062187842944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203528.708927:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187842944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203528.708936:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203528.708940:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203528.708943:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187842944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203528.708951:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187843072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203528.708983:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203528.708995:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203528/real 1788203528] req@ffff8e828793c700 x1875062187843072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203528.709008:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187843072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203528.709016:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203528.709020:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203528.709024:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187843072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203528.709030:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187843200:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203528.709051:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203528.709063:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203528/real 1788203528] req@ffff8e828793ce00 x1875062187843200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203528.709075:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187843200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203528.709084:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203528.709087:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203528.709091:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187843200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203529.412477:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203530.436551:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203531.460464:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203532.484473:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203533.508435:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203533.828406:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203533.828414:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203533.828418:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203533.828426:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4988 00000100:00080000:0.0:1788203533.828436:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203533.828442:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203533.828474:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187843328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203533.828495:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203533.828498:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203533.828499:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203533.828502:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4988 00000100:00080000:0.0:1788203533.828505:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203533.828508:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203533.828516:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203533.828530:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187843328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203533.828531:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187843456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203533.828559:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203533.828563:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203533.828566:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203533.828570:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4988 00000100:00080000:0.0:1788203533.828575:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203533.828579:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203533.828585:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203533.828595:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187843584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203533.828603:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788203533.828604:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203533.828609:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203533.828611:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203533.828615:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4988 00000100:00080000:0.0:1788203533.828619:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203533.828622:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203533.828627:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203533.828634:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187843712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203533.828641:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203533.828644:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203533.828645:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203533.828647:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4988 00000100:00080000:0.0:1788203533.828649:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203533.828652:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203533.828655:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203533.828659:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203533/real 1788203533] req@ffff8e828793ca80 x1875062187843328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203533.828664:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187843840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203533.828706:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187843328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203533.828715:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203533.828718:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203533.828722:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187843328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203533.828752:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187843456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203533.828773:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203533.828784:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203533/real 1788203533] req@ffff8e828793c000 x1875062187843456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203533.828793:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187843456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203533.828799:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203533.828801:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203533.828804:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187843456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203533.828810:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187843584:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203533.828830:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203533.828849:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203533/real 1788203533] req@ffff8e828793dc00 x1875062187843584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203533.828862:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187843584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203533.828873:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203533.828875:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203533.828877:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187843584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203533.828881:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187843712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203533.828898:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203533.828905:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203533/real 1788203533] req@ffff8e828793f480 x1875062187843712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203533.828913:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187843712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203533.828918:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203533.828920:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203533.828922:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187843712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203533.828925:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187843840:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203533.828935:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203533.828941:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203533/real 1788203533] req@ffff8e828793d500 x1875062187843840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203533.828948:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187843840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203533.828952:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203533.828954:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203533.828957:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187843840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203534.532481:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203535.556487:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203536.580466:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203537.604501:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203538.628448:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203538.948408:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203538.948417:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203538.948422:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203538.948431:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4994 00000100:00080000:0.0:1788203538.948440:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203538.948446:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203538.948504:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187843968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203538.948528:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203538.948531:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203538.948533:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203538.948536:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4994 00000100:00080000:0.0:1788203538.948545:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203538.948550:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203538.948559:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203538.948562:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187843968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203538.948572:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187844096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203538.948583:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203538.948586:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203538.948587:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203538.948591:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4994 00000100:00080000:0.0:1788203538.948594:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203538.948597:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203538.948601:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203538.948610:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187844224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203538.948617:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203538.948620:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203538.948623:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788203538.948624:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203538.948628:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4994 00000100:00080000:0.0:1788203538.948633:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203538.948636:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203538.948640:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203538.948647:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187844352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203538.948653:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203538.948655:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203538.948657:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203538.948660:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4994 00000100:00080000:0.0:1788203538.948662:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203538.948664:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203538.948667:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203538.948677:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187844480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203538.948677:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203538/real 1788203538] req@ffff8e828793e300 x1875062187843968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203538.948713:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187843968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203538.948724:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203538.948729:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203538.948734:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187843968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203538.948758:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187844096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203538.948781:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203538.948795:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203538/real 1788203538] req@ffff8e828793c380 x1875062187844096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203538.948805:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187844096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203538.948811:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203538.948814:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203538.948818:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187844096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203538.948822:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187844224:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203538.948835:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203538.948841:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203538/real 1788203538] req@ffff8e828793d880 x1875062187844224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203538.948849:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187844224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203538.948854:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203538.948856:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203538.948857:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187844224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203538.948861:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187844352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203538.948872:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203538.948877:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203538/real 1788203538] req@ffff8e828793fb80 x1875062187844352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203538.948884:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187844352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203538.948900:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203538.948903:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203538.948905:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187844352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203538.948908:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187844480:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203538.948921:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203538.948926:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203538/real 1788203538] req@ffff8e828793d500 x1875062187844480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203538.948933:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187844480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203538.948939:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203538.948940:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203538.948944:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187844480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203539.652503:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203540.676496:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203541.700527:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203542.724498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203543.748494:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203544.068453:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203544.068464:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203544.068469:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203544.068480:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 4999 00000100:00080000:2.0:1788203544.068491:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203544.068498:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203544.068537:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187844608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203544.068568:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203544.068573:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203544.068575:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203544.068580:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4999 00000100:00080000:2.0:1788203544.068584:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203544.068590:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203544.068599:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203544.068609:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187844608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203544.068617:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187844736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203544.068631:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203544.068634:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203544.068636:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203544.068640:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4999 00000100:00080000:2.0:1788203544.068644:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203544.068648:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203544.068653:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203544.068666:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187844864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203544.068675:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203544.068680:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1788203544.068682:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203544.068684:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203544.068691:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4999 00000100:00080000:2.0:1788203544.068695:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203544.068700:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203544.068706:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203544.068720:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187844992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203544.068731:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203544.068734:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203544.068736:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203544.068739:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 4999 00000100:00080000:2.0:1788203544.068743:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203544.068747:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203544.068753:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203544/real 1788203544] req@ffff8e82894e5c00 x1875062187844608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788203544.068754:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203544.068775:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187845120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203544.068782:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187844608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203544.068800:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203544.068805:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203544.068810:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187844608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203544.068835:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187844736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203544.068858:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203544.068872:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203544/real 1788203544] req@ffff8e82894e4380 x1875062187844736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203544.068885:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187844736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203544.068894:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203544.068898:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203544.068901:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187844736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203544.068908:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187844864:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203544.068929:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203544.068953:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203544/real 1788203544] req@ffff8e82894e5f80 x1875062187844864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203544.068966:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187844864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203544.068974:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203544.068978:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203544.068982:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187844864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203544.068989:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187844992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203544.069011:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203544.069024:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203544/real 1788203544] req@ffff8e82894e6d80 x1875062187844992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203544.069046:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187844992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203544.069059:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203544.069063:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203544.069067:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187844992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203544.069073:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187845120:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203544.069095:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203544.069107:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203544/real 1788203544] req@ffff8e82894e7100 x1875062187845120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203544.069120:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187845120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203544.069129:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203544.069133:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203544.069137:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187845120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203544.772558:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203545.796523:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203546.820498:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203547.844458:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203548.868473:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203549.188445:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203549.188457:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203549.188461:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203549.188472:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5004 00000100:00080000:3.0:1788203549.188484:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203549.188492:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203549.188535:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187845248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203549.188565:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203549.188568:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203549.188570:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203549.188573:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5004 00000100:00080000:3.0:1788203549.188576:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203549.188580:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203549.188590:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203549.188602:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187845376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203549.188612:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187845248:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788203549.188623:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203549.188628:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203549.188632:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203549.188639:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5004 00000100:00080000:3.0:1788203549.188644:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203549.188650:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203549.188658:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203549.188675:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187845504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203549.188688:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788203549.188700:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203549.188703:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203549.188708:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203549.188713:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5004 00000100:00080000:3.0:1788203549.188718:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203549.188722:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203549.188729:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203549.188757:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187845632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203549.188759:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203549/real 1788203549] req@ffff8e828953bb80 x1875062187845248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203549.188805:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203549.188808:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203549.188810:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203549.188814:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5004 00000100:00100000:2.0:1788203549.188820:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187845248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203549.188821:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203549.188830:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203549.188844:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203549.188855:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187845760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203549.188858:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203549.188875:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203549.188881:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187845248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203549.188906:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187845376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203549.188925:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203549.188935:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203549/real 1788203549] req@ffff8e8289538380 x1875062187845376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203549.188943:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187845376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203549.188949:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203549.188951:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203549.188953:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187845376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203549.188957:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187845504:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203549.188968:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203549.188974:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203549/real 1788203549] req@ffff8e8289539f80 x1875062187845504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203549.188981:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187845504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203549.188986:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203549.188987:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203549.188989:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187845504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203549.188993:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187845632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203549.189003:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203549.189009:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203549/real 1788203549] req@ffff8e8289539880 x1875062187845632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203549.189021:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187845632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203549.189026:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203549.189027:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203549.189029:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187845632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203549.189033:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187845760:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203549.189042:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203549.189048:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203549/real 1788203549] req@ffff8e8289539500 x1875062187845760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203549.189055:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187845760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203549.189061:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203549.189064:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203549.189068:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187845760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203549.892549:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203550.916443:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203551.940522:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203552.964481:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203553.988468:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203554.308415:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203554.308423:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203554.308427:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203554.308435:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5009 00000100:00080000:3.0:1788203554.308444:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203554.308457:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203554.308514:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187845888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203554.308539:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203554.308542:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203554.308544:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203554.308547:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5009 00000100:00080000:3.0:1788203554.308550:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203554.308553:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203554.308561:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203554.308572:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187846016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203554.308581:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203554.308584:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203554.308586:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203554.308589:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5009 00000100:00080000:3.0:1788203554.308591:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203554.308595:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203554.308598:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203554.308610:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187846144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203554.308613:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187845888:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788203554.308623:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203554.308628:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203554.308630:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203554.308633:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5009 00000100:00080000:3.0:1788203554.308635:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203554.308637:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203554.308641:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203554.308650:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187846272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203554.308658:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203554.308660:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203554.308661:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203554.308664:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5009 00000100:00080000:3.0:1788203554.308678:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203554.308681:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203554.308685:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203554.308694:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187846400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203554.308700:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203554.308774:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203554/real 1788203554] req@ffff8e828953b100 x1875062187845888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203554.308797:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187845888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203554.308827:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203554.308833:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203554.308840:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187845888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203554.308866:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187846016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203554.308891:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203554.308905:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203554/real 1788203554] req@ffff8e828953a680 x1875062187846016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203554.308935:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187846016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203554.308945:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203554.308949:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203554.308953:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187846016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203554.308978:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187846144:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203554.308999:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203554.309011:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203554/real 1788203554] req@ffff8e8289538a80 x1875062187846144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203554.309024:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187846144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203554.309034:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203554.309037:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203554.309041:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187846144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203554.309049:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187846272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203554.309069:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203554.309087:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203554/real 1788203554] req@ffff8e8289539500 x1875062187846272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203554.309101:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187846272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203554.309110:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203554.309114:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203554.309118:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187846272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203554.309124:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187846400:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203554.309144:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203554.309156:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203554/real 1788203554] req@ffff8e8289539880 x1875062187846400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203554.309169:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187846400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203554.309178:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203554.309182:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203554.309218:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187846400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203555.012477:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203556.036462:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203557.060487:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203558.084447:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203559.108491:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203559.428406:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203559.428424:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203559.428429:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203559.428439:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5014 00000100:00080000:3.0:1788203559.428450:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203559.428458:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203559.428498:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187846528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203559.428527:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203559.428531:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203559.428533:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203559.428538:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5014 00000100:00080000:3.0:1788203559.428542:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203559.428545:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203559.428553:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203559.428567:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187846656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203559.428574:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187846528:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788203559.428579:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203559.428583:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203559.428584:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203559.428588:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5014 00000100:00080000:3.0:1788203559.428590:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203559.428592:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203559.428603:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203559.428612:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187846784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203559.428620:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203559.428622:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203559.428624:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203559.428627:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5014 00000100:00080000:3.0:1788203559.428629:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203559.428632:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203559.428635:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788203559.428636:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203559.428646:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187846912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203559.428654:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203559.428656:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203559.428658:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203559.428660:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5014 00000100:00080000:3.0:1788203559.428662:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203559.428664:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203559.428668:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203559.428675:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187847040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203559.428686:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203559/real 1788203559] req@ffff8e8289539f80 x1875062187846528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203559.428707:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187846528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203559.428719:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203559.428724:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203559.428728:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187846528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203559.428747:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187846656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203559.428766:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203559.428776:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203559/real 1788203559] req@ffff8e8289538380 x1875062187846656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203559.428784:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187846656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203559.428790:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203559.428792:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203559.428795:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187846656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203559.428798:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187846784:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203559.428810:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203559.428816:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203559/real 1788203559] req@ffff8e828953bb80 x1875062187846784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203559.428823:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187846784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203559.428832:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203559.428835:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203559.428838:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187846784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203559.428844:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187846912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203559.428864:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203559.428875:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203559/real 1788203559] req@ffff8e8289539880 x1875062187846912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203559.428888:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187846912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203559.428896:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203559.428899:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203559.428901:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187846912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203559.428905:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187847040:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203559.428921:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203559.428930:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203559/real 1788203559] req@ffff8e8289539500 x1875062187847040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203559.428942:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187847040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203559.428950:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203559.428953:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203559.428957:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187847040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203560.132515:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203561.156498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203562.180490:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203563.204433:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203564.228500:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203564.548457:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203564.548467:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203564.548472:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203564.548482:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5019 00000100:00080000:0.0:1788203564.548491:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203564.548497:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203564.548528:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187847168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203564.548550:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203564.548554:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203564.548556:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203564.548559:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5019 00000100:00080000:0.0:1788203564.548561:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203564.548564:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203564.548571:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203564.548586:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187847296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203564.548594:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203564.548597:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203564.548598:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203564.548601:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5019 00000100:00080000:0.0:1788203564.548603:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203564.548608:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187847168:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203564.548609:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203564.548613:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203564.548623:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187847424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203564.548629:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203564.548632:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203564.548633:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203564.548635:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5019 00000100:00080000:0.0:1788203564.548637:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203564.548639:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203564.548643:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203564.548650:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187847552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203564.548657:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203564.548659:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203564.548660:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203564.548662:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5019 00000100:00080000:0.0:1788203564.548665:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203564.548667:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203564.548670:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203564.548676:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187847680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203564.548692:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203564.548737:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203564/real 1788203564] req@ffff8e828793f480 x1875062187847168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203564.548751:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187847168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203564.548759:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203564.548762:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203564.548765:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187847168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203564.548771:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187847296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203564.548788:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203564.548796:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203564/real 1788203564] req@ffff8e828793dc00 x1875062187847296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203564.548804:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187847296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203564.548809:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203564.548811:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203564.548814:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187847296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203564.548831:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187847424:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203564.548842:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203564.548849:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203564/real 1788203564] req@ffff8e828793c000 x1875062187847424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203564.548856:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187847424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203564.548861:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203564.548863:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203564.548865:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187847424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203564.548868:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187847552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203564.548879:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203564.548884:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203564/real 1788203564] req@ffff8e828793ca80 x1875062187847552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203564.548892:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187847552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203564.548896:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203564.548898:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203564.548900:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187847552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203564.548903:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187847680:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203564.548912:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203564.548918:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203564/real 1788203564] req@ffff8e828793ce00 x1875062187847680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203564.548925:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187847680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203564.548930:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203564.548932:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203564.548934:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187847680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203565.252490:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203566.276579:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203567.300544:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203568.324528:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203569.348504:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203569.668449:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203569.668462:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203569.668467:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203569.668480:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5024 00000100:00080000:0.0:1788203569.668494:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203569.668504:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203569.668556:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187847808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203569.668586:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203569.668589:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203569.668591:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203569.668594:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5024 00000100:00080000:0.0:1788203569.668598:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203569.668602:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203569.668613:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203569.668623:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187847936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203569.668641:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203569.668645:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203569.668650:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788203569.668652:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187847808:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203569.668659:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5024 00000100:00080000:0.0:1788203569.668671:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203569.668676:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203569.668682:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203569.668697:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187848064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203569.668704:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203569.668707:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203569.668709:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203569.668712:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5024 00000100:00080000:0.0:1788203569.668714:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203569.668718:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203569.668723:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203569.668733:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187848192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203569.668745:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203569.668748:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203569.668749:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203569.668751:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5024 00000100:00080000:0.0:1788203569.668754:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203569.668757:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203569.668760:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203569.668770:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187848320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203569.668797:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187847936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203569.668832:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187848064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203569.668852:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187848192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203569.668872:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187848320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203570.372440:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203571.396505:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203572.420487:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203572.740601:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788203572.740644:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788203572.740660:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203572.740700:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203572.740711:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203572.740720:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203572.740730:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:2.0:1788203572.740755:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203569/real 1788203572] req@ffff8e828793f100 x1875062187847936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203572.740783:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187847936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203572.740796:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203572.740801:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203572.740806:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187847936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203572.740814:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203569/real 1788203572] req@ffff8e828793e680 x1875062187848064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203572.740829:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187848064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203572.740845:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203572.740849:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203572.740852:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187848064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203572.740859:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203569/real 1788203572] req@ffff8e828793f800 x1875062187848192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203572.740911:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187848192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203572.740920:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203572.740924:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203572.740928:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187848192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203572.740935:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203569/real 1788203572] req@ffff8e828793ce00 x1875062187848320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203572.740947:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187848320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203572.740956:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203572.740959:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203572.740962:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187848320:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203572.740969:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203569/real 1788203572] req@ffff8e828793c700 x1875062187847808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203572.740980:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187847808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203572.740988:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203572.740991:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203572.740994:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187847808:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788203573.444515:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203574.468568:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203574.788542:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203574.788556:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203574.788563:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203574.788577:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5029 00000100:00080000:3.0:1788203574.788592:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203574.788602:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203574.788649:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187848448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203574.788701:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203574.788707:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203574.788709:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203574.788715:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5029 00000100:00080000:3.0:1788203574.788722:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203574.788730:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203574.788750:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203574.788762:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187848448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203574.788778:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187848576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203574.788800:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203574.788813:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203574.788820:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203574.788826:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5029 00000100:00080000:3.0:1788203574.788853:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203574.788860:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203574.788867:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788203574.788880:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203574.788888:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187848704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203574.788902:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203574.788907:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203574.788910:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203574.788917:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5029 00000100:00080000:3.0:1788203574.788923:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203574.788932:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203574.788941:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203574.788970:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187848832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203574.788971:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203574/real 1788203574] req@ffff8e8289538a80 x1875062187848448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203574.789012:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203574.789020:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203574.789023:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788203574.789038:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187848448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203574.789067:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203574.789072:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203574.789077:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187848448:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788203574.789104:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5029 00000100:00100000:0.0:1788203574.789112:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187848576:192.168.201.121@tcp:38:kworker.0 00000100:00080000:3.0:1788203574.789113:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203574.789128:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203574.789138:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203574.789161:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187848960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203574.789173:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203574.789197:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203574/real 1788203574] req@ffff8e828953a680 x1875062187848576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203574.789217:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187848576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203574.789228:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203574.789235:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203574.789239:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187848576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203574.789249:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187848704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203574.789273:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203574.789287:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203574/real 1788203574] req@ffff8e828953b100 x1875062187848704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203574.789302:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187848704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203574.789344:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203574.789351:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203574.789357:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187848704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203574.789366:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187848832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203574.789400:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203574.789428:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203574/real 1788203574] req@ffff8e8289539500 x1875062187848832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203574.789459:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187848832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203574.789475:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203574.789479:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203574.789482:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187848832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203574.789503:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187848960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203574.789526:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203574.789539:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203574/real 1788203574] req@ffff8e8289539880 x1875062187848960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203574.789551:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187848960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203574.789559:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203574.789563:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203574.789566:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187848960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203575.492474:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203576.517428:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203577.540454:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203578.564460:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203579.588454:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203579.908442:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203579.908450:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203579.908454:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203579.908463:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5034 00000100:00080000:0.0:1788203579.908471:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203579.908477:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203579.908509:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187849088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203579.908536:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203579.908541:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203579.908543:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203579.908548:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5034 00000100:00080000:0.0:1788203579.908553:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203579.908558:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203579.908568:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203579.908584:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187849216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203579.908594:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187849088:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203579.908598:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203579.908604:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203579.908606:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203579.908611:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5034 00000100:00080000:0.0:1788203579.908615:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203579.908619:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203579.908625:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203579.908638:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187849344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203579.908645:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203579.908647:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203579.908649:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203579.908651:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5034 00000100:00080000:0.0:1788203579.908655:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203579.908660:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:2.0:1788203579.908660:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788203579.908667:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203579.908674:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187849472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203579.908681:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203579.908684:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203579.908685:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203579.908687:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5034 00000100:00080000:0.0:1788203579.908689:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203579.908692:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203579.908695:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203579.908701:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187849600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203579.908719:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203579/real 1788203579] req@ffff8e828793ca80 x1875062187849088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203579.908735:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187849088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203579.908755:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203579.908759:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203579.908762:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187849088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203579.908781:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187849216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203579.908801:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203579.908810:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203579/real 1788203579] req@ffff8e828793c000 x1875062187849216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203579.908818:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187849216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203579.908824:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203579.908826:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203579.908828:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187849216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203579.908832:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187849344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203579.908844:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203579.908850:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203579/real 1788203579] req@ffff8e828793dc00 x1875062187849344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203579.908857:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187849344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203579.908873:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203579.908879:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203579.908882:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187849344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203579.908888:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187849472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203579.908910:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203579.908922:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203579/real 1788203579] req@ffff8e828793f480 x1875062187849472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203579.908935:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187849472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203579.908944:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203579.908947:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203579.908951:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187849472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203579.908957:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187849600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203579.908973:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203579.908980:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203579/real 1788203579] req@ffff8e828793d500 x1875062187849600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203579.908988:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187849600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203579.908994:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203579.908995:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203579.908997:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187849600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203580.612531:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203581.636624:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203582.660538:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203583.684531:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203584.708472:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203585.028442:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203585.028454:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203585.028458:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203585.028468:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5040 00000100:00080000:3.0:1788203585.028480:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203585.028487:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203585.028525:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187849728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203585.028555:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203585.028559:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203585.028560:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203585.028565:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5040 00000100:00080000:3.0:1788203585.028569:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203585.028573:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203585.028583:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203585.028597:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187849856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203585.028607:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187849728:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788203585.028613:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203585.028618:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203585.028620:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203585.028625:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5040 00000100:00080000:3.0:1788203585.028629:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203585.028633:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203585.028639:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203585.028652:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187849984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203585.028674:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203585.028678:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203585.028680:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203585.028684:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5040 00000800:00000100:2.0:1788203585.028689:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788203585.028690:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203585.028697:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203585.028704:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203585.028719:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187850112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203585.028729:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203585.028732:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788203585.028736:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203585/real 1788203585] req@ffff8e828953bb80 x1875062187849728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203585.028737:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203585.028749:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5040 00000100:00080000:3.0:1788203585.028755:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203585.028758:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187849728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203585.028764:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:2.0:1788203585.028774:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788203585.028775:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788203585.028781:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203585.028785:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187849728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203585.028793:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187850240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203585.028813:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187849856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203585.028843:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203585.028857:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203585/real 1788203585] req@ffff8e8289538380 x1875062187849856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203585.028866:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187849856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203585.028876:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203585.028878:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203585.028881:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187849856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203585.028897:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187849984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203585.028919:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203585.028931:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203585/real 1788203585] req@ffff8e8289539f80 x1875062187849984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203585.028945:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187849984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203585.028951:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203585.028953:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203585.028956:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187849984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203585.028962:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187850112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203585.028994:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203585.029007:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203585/real 1788203585] req@ffff8e8289539880 x1875062187850112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203585.029020:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187850112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203585.029029:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203585.029032:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203585.029036:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187850112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203585.029043:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187850240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203585.029064:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203585.029074:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203585/real 1788203585] req@ffff8e8289539500 x1875062187850240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203610 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203585.029087:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187850240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203610 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203585.029095:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203585.029099:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203585.029102:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187850240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203585.732490:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203586.756483:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203587.780458:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203588.804537:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203589.828463:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203590.148462:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203590.148473:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203590.148477:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203590.148488:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5045 00000100:00080000:2.0:1788203590.148500:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203590.148507:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203590.148545:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187850368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203590.148595:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203590.148600:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203590.148602:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203590.148607:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5045 00000100:00080000:2.0:1788203590.148613:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203590.148619:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203590.148629:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203590.148645:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187850368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203590.148646:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187850496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203590.148665:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203590.148669:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203590.148671:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203590.148675:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5045 00000100:00080000:2.0:1788203590.148679:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203590.148684:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203590.148691:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203590.148705:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187850624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203590.148716:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203590.148720:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203590.148724:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788203590.148730:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203590.148734:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5045 00000100:00080000:2.0:1788203590.148739:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203590.148745:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203590.148750:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203590.148763:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187850752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203590.148775:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203590.148779:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203590.148783:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788203590.148785:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203590/real 1788203590] req@ffff8e82894e6300 x1875062187850368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203590.148790:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5045 00000100:00080000:2.0:1788203590.148797:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203590.148805:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203590.148811:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203590.148819:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187850368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203590.148825:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187850880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203590.148840:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203590.148848:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203590.148854:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187850368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203590.148879:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187850496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203590.148906:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203590.148921:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203590/real 1788203590] req@ffff8e82894e7800 x1875062187850496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203590.148934:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187850496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203590.148943:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203590.148947:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203590.148950:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187850496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203590.148957:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187850624:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203590.148980:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203590.148993:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203590/real 1788203590] req@ffff8e82894e4a80 x1875062187850624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203590.149006:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187850624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203590.149014:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203590.149017:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203590.149021:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187850624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203590.149027:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187850752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203590.149047:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203590.149059:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203590/real 1788203590] req@ffff8e82894e4e00 x1875062187850752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203590.149080:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187850752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203590.149089:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203590.149092:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203590.149095:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187850752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203590.149102:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187850880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203590.149122:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203590.149135:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203590/real 1788203590] req@ffff8e82894e4000 x1875062187850880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203615 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203590.149146:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187850880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203615 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203590.149155:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203590.149159:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203590.149162:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187850880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203590.852541:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203591.876514:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203592.900506:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203593.924527:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203594.948520:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203595.268403:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203595.268411:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203595.268414:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203595.268421:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5050 00000100:00080000:2.0:1788203595.268429:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203595.268434:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203595.268461:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187851008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203595.268494:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203595.268498:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203595.268500:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203595.268504:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5050 00000100:00080000:2.0:1788203595.268507:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203595.268511:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203595.268520:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203595.268533:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187851136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203595.268544:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203595.268547:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203595.268549:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203595.268552:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5050 00000100:00080000:2.0:1788203595.268557:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203595.268561:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203595.268567:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203595.268576:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187851008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203595.268577:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187851264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203595.268598:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203595.268600:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203595.268602:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203595.268605:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5050 00000100:00080000:2.0:1788203595.268607:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203595.268611:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203595.268614:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203595.268621:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187851392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203595.268627:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203595.268629:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203595.268630:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203595.268632:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5050 00000100:00080000:2.0:1788203595.268635:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203595.268638:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203595.268642:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203595.268650:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187851520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203595.268668:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203595.268714:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203595/real 1788203595] req@ffff8e82894e6680 x1875062187851008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203595.268733:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187851008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203595.268753:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203595.268758:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203595.268762:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187851008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203595.268771:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187851136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203595.268796:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203595.268816:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203595/real 1788203595] req@ffff8e82894e5880 x1875062187851136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203595.268824:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187851136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203595.268830:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203595.268832:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203595.268834:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187851136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203595.268856:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187851264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203595.268869:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203595.268875:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203595/real 1788203595] req@ffff8e82894e6a00 x1875062187851264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203595.268882:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187851264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203595.268887:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203595.268889:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203595.268891:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187851264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203595.268895:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187851392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203595.268906:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203595.268911:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203595/real 1788203595] req@ffff8e82894e7480 x1875062187851392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203595.268917:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187851392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203595.268922:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203595.268924:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203595.268925:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187851392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203595.268928:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187851520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203595.268937:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203595.268942:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203595/real 1788203595] req@ffff8e82894e4000 x1875062187851520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203620 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203595.268948:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187851520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203620 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203595.268951:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203595.268953:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203595.268955:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187851520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203595.972506:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203596.996488:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203598.020566:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203599.044460:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203600.068482:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203600.388411:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203600.388421:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203600.388427:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203600.388438:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5055 00000100:00080000:3.0:1788203600.388448:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203600.388455:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203600.388490:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187851648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203600.388517:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203600.388521:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203600.388523:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203600.388527:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5055 00000100:00080000:3.0:1788203600.388531:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203600.388535:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203600.388543:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203600.388558:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187851776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203600.388569:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788203600.388571:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187851648:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788203600.388573:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203600.388577:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203600.388581:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5055 00000100:00080000:3.0:1788203600.388584:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203600.388586:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203600.388590:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203600.388599:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187851904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203600.388609:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203600.388611:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203600.388612:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203600.388615:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5055 00000100:00080000:3.0:1788203600.388616:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203600.388618:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203600.388621:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788203600.388624:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203600.388631:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187852032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203600.388638:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203600.388641:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203600.388642:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203600.388644:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5055 00000100:00080000:3.0:1788203600.388646:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203600.388648:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203600.388650:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203600.388658:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203600/real 1788203600] req@ffff8e828953b100 x1875062187851648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203600.388659:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187852160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203600.388692:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187851648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203600.388700:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203600.388703:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203600.388705:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187851648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203600.388721:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187851776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203600.388733:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203600.388740:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203600/real 1788203600] req@ffff8e828953a680 x1875062187851776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203600.388751:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187851776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203600.388758:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203600.388761:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203600.388764:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187851776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203600.388770:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187851904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203600.388788:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203600.388797:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203600/real 1788203600] req@ffff8e8289538a80 x1875062187851904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203600.388804:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187851904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203600.388813:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203600.388815:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203600.388816:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187851904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203600.388820:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187852032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203600.388831:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203600.388836:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203600/real 1788203600] req@ffff8e8289539500 x1875062187852032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203600.388842:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187852032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203600.388847:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203600.388849:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203600.388852:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187852032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203600.388857:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187852160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203600.388872:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203600.388882:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203600/real 1788203600] req@ffff8e8289539880 x1875062187852160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203600.388892:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187852160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203600.388899:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203600.388902:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203600.388905:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187852160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203601.092466:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203602.116484:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203603.140474:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203604.164473:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203605.188492:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203605.508491:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203605.508503:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203605.508509:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203605.508522:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5060 00000100:00080000:0.0:1788203605.508536:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203605.508545:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203605.508604:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187852288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203605.508640:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203605.508644:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203605.508647:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203605.508652:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5060 00000100:00080000:0.0:1788203605.508657:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203605.508663:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203605.508675:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203605.508690:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187852288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203605.508691:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187852416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203605.508706:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203605.508709:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203605.508710:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203605.508713:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5060 00000100:00080000:0.0:1788203605.508717:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203605.508721:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203605.508725:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203605.508743:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187852544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203605.508750:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203605.508753:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203605.508754:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203605.508757:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5060 00000100:00080000:0.0:1788203605.508760:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203605.508763:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203605.508767:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788203605.508773:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203605.508779:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187852672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203605.508791:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203605.508795:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203605.508797:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203605.508802:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5060 00000100:00080000:0.0:1788203605.508807:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203605.508812:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203605.508817:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203605.508831:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187852800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203605.508836:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203605/real 1788203605] req@ffff8e828793fb80 x1875062187852288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203605.508859:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187852288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203605.508869:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203605.508873:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203605.508876:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187852288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203605.508900:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187852416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203605.508919:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203605.508928:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203605/real 1788203605] req@ffff8e828793d880 x1875062187852416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203605.508937:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187852416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203605.508942:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203605.508945:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203605.508948:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187852416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203605.508952:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187852544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203605.508966:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203605.508972:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203605/real 1788203605] req@ffff8e828793c380 x1875062187852544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203605.508980:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187852544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203605.508985:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203605.508987:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203605.508989:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187852544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203605.508992:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187852672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203605.509004:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203605.509010:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203605/real 1788203605] req@ffff8e828793e300 x1875062187852672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203605.509018:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187852672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203605.509023:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203605.509024:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203605.509026:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187852672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203605.509029:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187852800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203605.509052:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203605.509059:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203605/real 1788203605] req@ffff8e828793d500 x1875062187852800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203605.509067:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187852800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203605.509073:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203605.509076:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203605.509080:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187852800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203606.212544:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203607.236513:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203608.260510:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203609.284485:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203610.308501:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203610.628437:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203610.628452:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203610.628459:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203610.628472:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5065 00000100:00080000:0.0:1788203610.628487:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203610.628497:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203610.628558:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187852928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203610.628591:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203610.628595:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203610.628597:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203610.628603:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5065 00000100:00080000:0.0:1788203610.628608:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203610.628613:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203610.628624:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203610.628642:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187853056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203610.628655:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187852928:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203610.628656:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203610.628665:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203610.628668:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203610.628673:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5065 00000100:00080000:0.0:1788203610.628676:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203610.628680:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203610.628686:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203610.628700:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187853184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203610.628711:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203610.628715:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203610.628717:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203610.628723:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5065 00000100:00080000:0.0:1788203610.628727:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203610.628732:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203610.628738:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788203610.628748:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203610.628754:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187853312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203610.628767:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203610.628771:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203610.628773:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203610.628777:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5065 00000100:00080000:0.0:1788203610.628782:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203610.628787:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203610.628792:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203610.628800:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203610/real 1788203610] req@ffff8e828793f480 x1875062187852928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203610.628804:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187853440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203610.628860:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187852928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203610.628875:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203610.628879:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203610.628883:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187852928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203610.628910:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187853056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203610.628934:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203610.628953:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203610/real 1788203610] req@ffff8e828793dc00 x1875062187853056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203610.628963:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187853056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203610.628976:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203610.628979:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203610.628982:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187853056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203610.628986:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187853184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203610.629002:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203610.629012:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203610/real 1788203610] req@ffff8e828793c000 x1875062187853184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203610.629020:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187853184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203610.629025:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203610.629027:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203610.629029:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187853184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203610.629034:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187853312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203610.629046:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203610.629052:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203610/real 1788203610] req@ffff8e828793ca80 x1875062187853312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203610.629059:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187853312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203610.629064:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203610.629066:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203610.629069:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187853312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203610.629073:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187853440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203610.629083:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203610.629089:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203610/real 1788203610] req@ffff8e828793c700 x1875062187853440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203610.629096:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187853440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203610.629102:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203610.629103:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203610.629105:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187853440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203611.332440:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203612.356505:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203613.380569:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203614.404530:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203615.428523:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203615.748433:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203615.748448:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203615.748464:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203615.748481:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5070 00000100:00080000:1.0:1788203615.748507:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203615.748516:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203615.748562:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187853568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203615.748595:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203615.748598:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203615.748600:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203615.748605:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5070 00000100:00080000:1.0:1788203615.748608:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203615.748612:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203615.748622:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203615.748633:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187853568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203615.748638:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187853696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203615.748653:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203615.748656:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203615.748657:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203615.748660:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5070 00000100:00080000:1.0:1788203615.748663:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203615.748665:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203615.748668:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203615.748678:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187853824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203615.748684:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203615.748687:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203615.748688:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203615.748691:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5070 00000100:00080000:1.0:1788203615.748694:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203615.748697:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203615.748700:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203615.748708:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187853952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203615.748716:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788203615.748719:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203615.748723:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203615.748724:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203615.748728:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5070 00000100:00080000:1.0:1788203615.748732:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203615.748735:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203615.748739:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203615.748746:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187854080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203615.748779:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203615/real 1788203615] req@ffff8e828d720380 x1875062187853568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203615.748798:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187853568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203615.748808:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203615.748811:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203615.748815:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187853568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203615.748838:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187853696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203615.748860:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203615.748875:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203615/real 1788203615] req@ffff8e828d722300 x1875062187853696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203615.748889:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187853696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203615.748900:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203615.748903:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203615.748907:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187853696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203615.748913:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187853824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203615.748936:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203615.748948:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203615/real 1788203615] req@ffff8e828d722d80 x1875062187853824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203615.748962:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187853824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203615.748978:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203615.748981:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203615.748985:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187853824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203615.748991:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187853952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203615.749012:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203615.749024:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203615/real 1788203615] req@ffff8e828d721180 x1875062187853952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203615.749051:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187853952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203615.749058:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203615.749060:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203615.749062:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187853952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203615.749067:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187854080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203615.749094:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203615.749101:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203615/real 1788203615] req@ffff8e828d720a80 x1875062187854080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203615.749108:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187854080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203615.749113:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203615.749115:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203615.749117:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187854080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203616.452476:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203617.476579:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203618.500539:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203619.524524:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203620.548487:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203620.868437:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203620.868445:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203620.868448:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203620.868457:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5075 00000100:00080000:3.0:1788203620.868467:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203620.868474:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203620.868506:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187854208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203620.868530:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203620.868532:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203620.868534:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203620.868536:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5075 00000100:00080000:3.0:1788203620.868538:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203620.868541:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203620.868547:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203620.868558:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187854336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203620.868565:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203620.868568:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203620.868569:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203620.868572:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5075 00000100:00080000:3.0:1788203620.868574:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203620.868577:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203620.868582:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187854208:192.168.201.121@tcp:250:kworker.0 00000008:00100000:3.0:1788203620.868582:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203620.868592:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187854464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203620.868599:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203620.868602:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203620.868603:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203620.868606:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5075 00000100:00080000:3.0:1788203620.868608:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203620.868610:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203620.868613:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203620.868623:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187854592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203620.868633:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203620.868635:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203620.868637:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788203620.868643:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788203620.868644:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5075 00000100:00080000:3.0:1788203620.868651:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203620.868657:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203620.868663:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203620.868676:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187854720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203620.868695:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203620/real 1788203620] req@ffff8e8289539f80 x1875062187854208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203620.868710:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187854208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203620.868718:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203620.868721:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203620.868725:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187854208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203620.868734:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187854336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203620.868752:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203620.868761:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203620/real 1788203620] req@ffff8e8289538380 x1875062187854336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203620.868769:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187854336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203620.868774:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203620.868777:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203620.868779:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187854336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203620.868797:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187854464:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203620.868808:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203620.868814:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203620/real 1788203620] req@ffff8e828953bb80 x1875062187854464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203620.868822:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187854464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203620.868827:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203620.868829:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203620.868831:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187854464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203620.868834:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187854592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203620.868845:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203620.868851:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203620/real 1788203620] req@ffff8e8289539880 x1875062187854592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203620.868858:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187854592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203620.868863:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203620.868865:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203620.868875:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187854592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203620.868879:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187854720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203620.868890:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203620.868896:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203620/real 1788203620] req@ffff8e8289539500 x1875062187854720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203620.868903:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187854720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203620.868908:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203620.868910:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203620.868913:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187854720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203621.572557:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203622.596490:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203623.620498:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203624.644514:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203625.668575:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203625.988463:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203625.988477:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203625.988483:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203625.988497:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5081 00000100:00080000:0.0:1788203625.988512:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203625.988521:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203625.988571:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187854848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203625.988607:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203625.988611:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203625.988614:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203625.988619:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5081 00000100:00080000:0.0:1788203625.988623:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203625.988626:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203625.988639:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203625.988652:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187854976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203625.988663:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187854848:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203625.988667:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203625.988674:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203625.988677:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203625.988683:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5081 00000100:00080000:0.0:1788203625.988687:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203625.988691:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203625.988696:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203625.988707:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187855104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203625.988718:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203625.988723:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203625.988725:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203625.988729:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5081 00000100:00080000:0.0:1788203625.988744:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203625.988748:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203625.988754:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788203625.988756:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203625.988778:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187855232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203625.988790:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203625.988794:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203625.988797:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203625.988802:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5081 00000100:00080000:0.0:1788203625.988806:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203625.988810:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203625/real 1788203625] req@ffff8e828793ce00 x1875062187854848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203625.988813:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203625.988821:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203625.988833:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187854848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203625.988838:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187855360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203625.988874:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203625.988880:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203625.988884:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187854848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203625.988910:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187854976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203625.988928:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203625.988937:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203625/real 1788203625] req@ffff8e828793f800 x1875062187854976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203625.988960:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187854976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203625.988966:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203625.988969:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203625.988971:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187854976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203625.988976:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187855104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203625.988991:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203625.988998:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203625/real 1788203625] req@ffff8e828793e680 x1875062187855104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203625.989006:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187855104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203625.989012:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203625.989013:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203625.989015:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187855104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203625.989020:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187855232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203625.989032:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203625.989038:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203625/real 1788203625] req@ffff8e828793f100 x1875062187855232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203625.989046:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187855232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203625.989051:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203625.989053:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203625.989055:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187855232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203625.989058:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187855360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203625.989072:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203625.989082:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203625/real 1788203625] req@ffff8e828793c700 x1875062187855360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203625.989095:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187855360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203625.989104:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203625.989107:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203625.989111:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187855360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203626.692503:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203627.716553:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203628.740528:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203629.764566:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203630.788678:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203631.108405:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203631.108418:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203631.108423:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203631.108435:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5086 00000100:00080000:0.0:1788203631.108449:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203631.108457:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203631.108503:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187855488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203631.108532:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203631.108536:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203631.108539:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203631.108545:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5086 00000100:00080000:0.0:1788203631.108548:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203631.108551:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203631.108561:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203631.108569:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187855616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203631.108577:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203631.108580:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203631.108583:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203631.108587:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5086 00000100:00080000:0.0:1788203631.108590:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203631.108592:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203631.108596:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203631.108602:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187855744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203631.108611:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203631.108615:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788203631.108615:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187855488:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203631.108622:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203631.108635:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5086 00000100:00080000:0.0:1788203631.108639:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203631.108641:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203631.108645:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203631.108664:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187855872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203631.108671:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203631.108674:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203631.108675:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203631.108677:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5086 00000100:00080000:0.0:1788203631.108682:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203631.108687:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203631.108690:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203631.108697:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187856000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203631.108706:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203631.108759:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203631/real 1788203631] req@ffff8e828793ca80 x1875062187855488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203631.108780:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187855488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203631.108788:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203631.108792:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203631.108797:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187855488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203631.108806:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187855616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203631.108830:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203631.108844:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203631/real 1788203631] req@ffff8e828793c000 x1875062187855616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203631.108871:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187855616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203631.108878:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203631.108880:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203631.108882:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187855616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203631.108903:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187855744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203631.108919:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203631.108927:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203631/real 1788203631] req@ffff8e828793dc00 x1875062187855744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203631.108934:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187855744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203631.108944:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203631.108946:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203631.108947:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187855744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203631.108951:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187855872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203631.108962:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203631.108968:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203631/real 1788203631] req@ffff8e828793f480 x1875062187855872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203631.108975:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187855872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203631.108980:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203631.108982:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203631.108984:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187855872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203631.108987:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187856000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203631.108997:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203631.109002:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203631/real 1788203631] req@ffff8e828793d500 x1875062187856000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203656 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203631.109008:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187856000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203656 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203631.109013:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203631.109015:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203631.109017:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187856000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203631.812508:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203632.836590:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203633.860538:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203634.884531:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203635.908471:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203636.228455:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203636.228465:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203636.228469:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203636.228479:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5091 00000100:00080000:1.0:1788203636.228490:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203636.228497:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203636.228534:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187856128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203636.228563:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203636.228567:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203636.228569:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203636.228573:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5091 00000100:00080000:1.0:1788203636.228576:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203636.228581:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203636.228591:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203636.228604:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187856256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203636.228607:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187856128:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788203636.228623:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203636.228627:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203636.228629:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203636.228634:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5091 00000100:00080000:1.0:1788203636.228638:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203636.228643:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203636.228649:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203636.228662:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187856384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203636.228673:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203636.228677:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203636.228679:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203636.228683:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5091 00000100:00080000:1.0:1788203636.228687:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203636.228691:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203636.228696:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203636.228707:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187856512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203636.228718:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203636.228722:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788203636.228724:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187856256:192.168.201.121@tcp:38:kworker.0 00000100:00080000:1.0:1788203636.228725:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203636.228734:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5091 00000100:00080000:1.0:1788203636.228738:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203636.228742:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203636.228747:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203636.228760:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187856640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203636.228784:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187856384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203636.228806:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187856512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203636.228830:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187856640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203636.932474:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203637.956447:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203638.980470:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203639.300615:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788203639.300662:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788203639.300677:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203639.300745:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:0.0:1788203639.300757:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203636/real 1788203639] req@ffff8e828d720a80 x1875062187856128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1788203639.300759:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203639.300772:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203639.300785:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:0.0:1788203639.300790:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187856128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203639.300802:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203639.300807:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203639.300812:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187856128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203639.300831:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203636/real 1788203639] req@ffff8e828d720380 x1875062187856640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203639.300850:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187856640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203639.300857:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203639.300860:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203639.300863:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187856640:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203639.300869:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203636/real 1788203639] req@ffff8e828d721180 x1875062187856256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203639.300880:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187856256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203639.300887:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203639.300890:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203639.300893:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187856256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203639.300900:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203636/real 1788203639] req@ffff8e828d722d80 x1875062187856384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203639.300910:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187856384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203639.300918:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203639.300921:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203639.300925:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187856384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203639.300931:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203636/real 1788203639] req@ffff8e828d722300 x1875062187856512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203661 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203639.300943:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187856512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203661 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203639.300950:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203639.300954:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203639.300956:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187856512:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203640.004487:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203641.028488:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203641.348410:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203641.348418:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203641.348421:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203641.348429:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5096 00000100:00080000:1.0:1788203641.348438:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203641.348444:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203641.348470:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187856768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203641.348487:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203641.348490:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203641.348491:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203641.348494:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5096 00000100:00080000:1.0:1788203641.348496:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203641.348499:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203641.348506:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203641.348518:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187856768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203641.348519:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187856896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203641.348532:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203641.348537:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203641.348538:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203641.348541:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5096 00000100:00080000:1.0:1788203641.348545:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203641.348555:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203641.348561:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203641.348574:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187857024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203641.348590:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788203641.348594:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788203641.348596:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203641.348599:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203641.348605:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5096 00000100:00080000:1.0:1788203641.348611:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203641.348614:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203641.348618:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203641.348627:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203641/real 1788203641] req@ffff8e828d720e00 x1875062187856768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203641.348629:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187857152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203641.348663:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187856768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203641.348664:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203641.348671:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203641.348697:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203641.348701:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:1.0:1788203641.348703:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788203641.348705:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187856768:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788203641.348711:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5096 00000100:00080000:1.0:1788203641.348717:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203641.348721:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203641.348727:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203641.348735:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187856896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203641.348740:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187857280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203641.348763:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203641.348780:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203641/real 1788203641] req@ffff8e828d723800 x1875062187856896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203641.348791:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187856896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203641.348804:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203641.348807:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203641.348810:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187856896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203641.348815:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187857024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203641.348834:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203641.348844:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203641/real 1788203641] req@ffff8e828d721500 x1875062187857024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203641.348854:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187857024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203641.348860:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203641.348862:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203641.348865:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187857024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203641.348870:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187857152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203641.348887:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203641.348909:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203641/real 1788203641] req@ffff8e828d720000 x1875062187857152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203641.348921:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187857152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203641.348927:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203641.348930:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203641.348933:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187857152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203641.348948:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187857280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203641.348964:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203641.348974:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203641/real 1788203641] req@ffff8e828d722680 x1875062187857280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203641.348984:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187857280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203641.348991:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203641.348993:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203641.348996:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187857280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203642.052454:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203643.076533:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203644.100432:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203645.124468:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203646.148498:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203646.468402:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203646.468410:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203646.468415:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203646.468426:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5101 00000100:00080000:1.0:1788203646.468436:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203646.468443:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203646.468478:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187857408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203646.468503:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203646.468506:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203646.468507:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203646.468510:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5101 00000100:00080000:1.0:1788203646.468513:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203646.468516:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203646.468529:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203646.468543:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187857408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203646.468547:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187857536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203646.468561:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203646.468563:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203646.468565:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203646.468568:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5101 00000100:00080000:1.0:1788203646.468571:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203646.468575:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203646.468581:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203646.468606:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187857664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203646.468618:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203646.468620:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203646.468624:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788203646.468625:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788203646.468633:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5101 00000100:00080000:1.0:1788203646.468638:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203646.468642:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203646.468648:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203646.468660:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187857792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203646.468668:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203646/real 1788203646] req@ffff8e828d722680 x1875062187857408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203646.468671:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203646.468676:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203646.468677:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203646.468682:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5101 00000100:00080000:1.0:1788203646.468687:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203646.468693:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203646.468700:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203646.468707:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187857408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203646.468712:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187857920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203646.468731:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203646.468737:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203646.468742:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187857408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203646.468766:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187857536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203646.468790:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203646.468802:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203646/real 1788203646] req@ffff8e828d720000 x1875062187857536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203646.468814:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187857536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203646.468823:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203646.468826:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203646.468830:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187857536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203646.468836:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187857664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203646.468859:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203646.468870:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203646/real 1788203646] req@ffff8e828d721500 x1875062187857664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203646.468883:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187857664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203646.468891:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203646.468905:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203646.468909:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187857664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203646.468919:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187857792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203646.468943:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203646.468960:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203646/real 1788203646] req@ffff8e828d723800 x1875062187857792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203646.468973:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187857792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203646.468982:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203646.468986:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203646.468992:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187857792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203646.468998:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187857920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203646.469015:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203646.469025:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203646/real 1788203646] req@ffff8e828d720e00 x1875062187857920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203646.469035:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187857920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203646.469042:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203646.469045:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203646.469048:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187857920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203647.172499:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203648.196507:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203649.220534:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203650.244489:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203651.268453:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203651.588416:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203651.588426:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203651.588431:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203651.588439:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5106 00000100:00080000:3.0:1788203651.588449:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203651.588457:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203651.588492:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187858048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203651.588532:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203651.588536:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203651.588538:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203651.588542:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5106 00000100:00080000:3.0:1788203651.588546:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203651.588550:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203651.588560:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203651.588575:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187858176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203651.588585:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203651.588589:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203651.588591:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203651.588594:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5106 00000100:00080000:3.0:1788203651.588599:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203651.588605:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203651.588613:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187858048:192.168.201.121@tcp:250:kworker.0 00000008:00100000:3.0:1788203651.588614:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203651.588637:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187858304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203651.588652:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203651.588655:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203651.588657:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203651.588662:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5106 00000100:00080000:3.0:1788203651.588666:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203651.588670:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203651.588675:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203651.588686:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187858432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203651.588694:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203651.588697:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203651.588699:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203651.588702:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5106 00000100:00080000:3.0:1788203651.588706:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203651.588710:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:2.0:1788203651.588714:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:3.0:1788203651.588719:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203651.588734:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187858560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203651.588789:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203651/real 1788203651] req@ffff8e8289538a80 x1875062187858048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203651.588820:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187858048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203651.588832:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203651.588836:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203651.588839:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187858048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203651.588845:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187858176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203651.588866:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203651.588876:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203651/real 1788203651] req@ffff8e828953a680 x1875062187858176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203651.588883:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187858176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203651.588889:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203651.588891:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203651.588893:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187858176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203651.588915:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187858304:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203651.588925:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203651.588931:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203651/real 1788203651] req@ffff8e828953b100 x1875062187858304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203651.588937:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187858304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203651.588942:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203651.588943:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203651.588945:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187858304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203651.588948:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187858432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203651.588958:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203651.588964:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203651/real 1788203651] req@ffff8e8289539500 x1875062187858432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203651.588970:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187858432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203651.588975:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203651.588977:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203651.588979:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187858432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203651.588982:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187858560:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203651.588991:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203651.588996:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203651/real 1788203651] req@ffff8e8289539880 x1875062187858560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203651.589001:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187858560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203651.589005:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203651.589007:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203651.589009:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187858560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203652.292472:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203653.316542:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203654.340519:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203655.364499:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203656.388493:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203656.708401:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203656.708424:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203656.708427:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203656.708436:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5111 00000100:00080000:3.0:1788203656.708445:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203656.708452:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203656.708483:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187858688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203656.708506:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203656.708509:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203656.708510:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203656.708513:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5111 00000100:00080000:3.0:1788203656.708515:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203656.708517:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203656.708524:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203656.708534:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187858816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203656.708541:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203656.708544:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203656.708545:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203656.708547:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5111 00000100:00080000:3.0:1788203656.708549:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203656.708553:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203656.708555:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203656.708561:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187858944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203656.708570:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187858688:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788203656.708576:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203656.708581:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203656.708582:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203656.708585:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5111 00000100:00080000:3.0:1788203656.708587:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203656.708589:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203656.708592:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203656.708601:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187859072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203656.708608:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203656.708610:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203656.708611:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203656.708613:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5111 00000100:00080000:3.0:1788203656.708627:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203656.708630:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203656.708633:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203656.708641:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187859200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203656.708647:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203656.708710:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203656/real 1788203656] req@ffff8e828953bb80 x1875062187858688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203656.708731:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187858688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203656.708743:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203656.708748:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203656.708752:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187858688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203656.708761:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187858816:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203656.708784:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203656.708797:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203656/real 1788203656] req@ffff8e8289538380 x1875062187858816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203656.708810:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187858816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203656.708819:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203656.708822:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203656.708826:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187858816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203656.708850:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187858944:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203656.708871:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203656.708882:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203656/real 1788203656] req@ffff8e8289539f80 x1875062187858944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203656.708895:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187858944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203656.708904:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203656.708907:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203656.708911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187858944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203656.708918:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187859072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203656.708938:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203656.708956:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203656/real 1788203656] req@ffff8e8289539880 x1875062187859072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203656.708971:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187859072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203656.708980:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203656.708983:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203656.708986:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187859072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203656.708993:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187859200:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203656.709013:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203656.709024:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203656/real 1788203656] req@ffff8e8289539500 x1875062187859200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203656.709037:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187859200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203656.709045:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203656.709047:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203656.709051:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187859200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203657.412493:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203658.436507:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203659.460531:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203660.484517:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203661.508506:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203661.828592:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203661.828599:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203661.828602:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203661.828610:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5116 00000100:00080000:3.0:1788203661.828619:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203661.828626:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203661.828661:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187859328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203661.828688:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203661.828692:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203661.828695:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203661.828699:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5116 00000100:00080000:3.0:1788203661.828702:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203661.828705:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203661.828714:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203661.828728:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187859456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203661.828737:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203661.828740:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203661.828741:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203661.828744:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5116 00000100:00080000:3.0:1788203661.828747:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203661.828749:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187859328:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788203661.828750:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203661.828765:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203661.828779:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187859584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203661.828790:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203661.828794:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203661.828796:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203661.828800:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5116 00000100:00080000:3.0:1788203661.828807:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203661.828813:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203661.828818:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788203661.828831:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203661.828832:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187859712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203661.828846:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203661.828849:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203661.828852:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203661.828856:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5116 00000100:00080000:3.0:1788203661.828860:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203661.828865:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203661.828870:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203661.828877:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203661/real 1788203661] req@ffff8e828953b100 x1875062187859328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203661.828885:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187859840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203661.828929:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187859328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203661.828948:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203661.828952:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203661.828957:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187859328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203661.828980:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187859456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203661.829005:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203661.829019:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203661/real 1788203661] req@ffff8e828953a680 x1875062187859456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203661.829032:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187859456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203661.829041:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203661.829045:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203661.829048:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187859456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203661.829055:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187859584:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203661.829074:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203661.829085:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203661/real 1788203661] req@ffff8e8289538a80 x1875062187859584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203661.829096:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187859584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203661.829104:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203661.829108:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203661.829111:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187859584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203661.829118:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187859712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203661.829137:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203661.829149:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203661/real 1788203661] req@ffff8e8289539500 x1875062187859712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203661.829161:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187859712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203661.829182:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203661.829186:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203661.829190:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187859712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203661.829196:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187859840:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203661.829215:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203661.829225:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203661/real 1788203661] req@ffff8e8289539880 x1875062187859840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203661.829237:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187859840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203661.829245:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203661.829248:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203661.829252:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187859840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203662.532493:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203663.556466:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203664.580486:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203665.604458:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203666.628514:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203666.948449:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203666.948457:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203666.948461:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203666.948470:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5122 00000100:00080000:2.0:1788203666.948480:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203666.948486:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203666.948523:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187859968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203666.948554:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203666.948557:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203666.948558:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203666.948562:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5122 00000100:00080000:2.0:1788203666.948565:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203666.948568:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203666.948576:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203666.948587:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187860096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203666.948595:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203666.948600:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788203666.948601:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187859968:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203666.948602:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203666.948610:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5122 00000100:00080000:2.0:1788203666.948613:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203666.948615:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203666.948618:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203666.948627:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187860224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203666.948642:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203666.948646:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203666.948649:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203666.948653:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5122 00000100:00080000:2.0:1788203666.948658:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203666.948663:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:0.0:1788203666.948675:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:2.0:1788203666.948680:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203666.948697:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187860352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203666.948706:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203666.948710:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203666.948712:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203666.948716:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5122 00000100:00080000:2.0:1788203666.948721:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788203666.948726:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203666/real 1788203666] req@ffff8e82894e4e00 x1875062187859968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203666.948729:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203666.948740:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203666.948748:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187859968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203666.948756:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187860480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203666.948761:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203666.948770:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203666.948777:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187859968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203666.948800:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187860096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203666.948819:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203666.948831:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203666/real 1788203666] req@ffff8e82894e4a80 x1875062187860096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203666.948844:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187860096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203666.948868:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203666.948873:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203666.948877:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187860096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203666.948884:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187860224:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203666.948901:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203666.948910:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203666/real 1788203666] req@ffff8e82894e7800 x1875062187860224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203666.948918:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187860224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203666.948924:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203666.948926:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203666.948928:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187860224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203666.948933:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187860352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203666.948947:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203666.948953:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203666/real 1788203666] req@ffff8e82894e6300 x1875062187860352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203666.948965:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187860352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203666.948973:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203666.948977:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203666.948980:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187860352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203666.948986:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187860480:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203666.949007:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203666.949018:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203666/real 1788203666] req@ffff8e82894e7100 x1875062187860480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203666.949026:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187860480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203666.949032:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203666.949034:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203666.949036:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187860480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203667.652510:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203668.676483:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203669.700471:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203670.724437:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203671.748481:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203672.068389:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203672.068397:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203672.068400:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203672.068407:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5127 00000100:00080000:3.0:1788203672.068417:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203672.068422:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203672.068451:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187860608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203672.068476:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203672.068479:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203672.068480:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203672.068483:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5127 00000100:00080000:3.0:1788203672.068485:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203672.068487:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203672.068494:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203672.068504:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187860736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203672.068511:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203672.068513:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203672.068515:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788203672.068519:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187860608:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788203672.068520:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5127 00000100:00080000:3.0:1788203672.068525:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203672.068528:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203672.068532:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203672.068540:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187860864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203672.068546:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203672.068548:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203672.068549:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203672.068551:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5127 00000100:00080000:3.0:1788203672.068555:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203672.068557:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203672.068559:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203672.068566:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187860992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203672.068572:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788203672.068578:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203672.068582:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203672.068583:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203672.068586:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5127 00000100:00080000:3.0:1788203672.068589:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203672.068592:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203672.068595:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203672.068602:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187861120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203672.068624:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203672/real 1788203672] req@ffff8e8289539f80 x1875062187860608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203672.068641:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187860608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203672.068648:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203672.068651:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203672.068653:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187860608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203672.068670:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187860736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203672.068684:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203672.068692:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203672/real 1788203672] req@ffff8e8289538380 x1875062187860736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203672.068699:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187860736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203672.068704:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203672.068705:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203672.068707:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187860736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203672.068710:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187860864:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203672.068719:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203672.068724:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203672/real 1788203672] req@ffff8e828953bb80 x1875062187860864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203672.068730:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187860864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203672.068739:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203672.068741:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203672.068742:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187860864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203672.068745:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187860992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203672.068767:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203672.068772:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203672/real 1788203672] req@ffff8e8289539880 x1875062187860992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203672.068778:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187860992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203672.068783:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203672.068785:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203672.068786:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187860992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203672.068789:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187861120:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203672.068798:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203672.068803:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203672/real 1788203672] req@ffff8e8289539500 x1875062187861120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203672.068809:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187861120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203672.068813:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203672.068814:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203672.068816:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187861120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203672.772477:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203673.796418:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203674.821584:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203675.844469:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203676.868490:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203677.188452:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203677.188459:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203677.188462:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203677.188469:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5132 00000100:00080000:0.0:1788203677.188478:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203677.188483:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203677.188521:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187861248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203677.188541:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203677.188544:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203677.188545:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203677.188548:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5132 00000100:00080000:0.0:1788203677.188550:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203677.188552:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203677.188558:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203677.188566:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187861376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203677.188572:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203677.188574:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203677.188575:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203677.188577:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5132 00000100:00080000:0.0:1788203677.188580:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203677.188582:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203677.188585:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203677.188587:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187861248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203677.188593:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187861504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203677.188599:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203677.188601:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203677.188602:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203677.188605:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5132 00000100:00080000:0.0:1788203677.188606:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203677.188608:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203677.188611:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203677.188618:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187861632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203677.188623:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203677.188625:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203677.188626:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203677.188629:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5132 00000100:00080000:0.0:1788203677.188633:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203677.188637:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203677.188641:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788203677.188642:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203677.188658:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187861760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203677.188676:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203677/real 1788203677] req@ffff8e828793e300 x1875062187861248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203677.188699:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187861248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203677.188709:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203677.188713:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203677.188717:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187861248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203677.188724:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187861376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203677.188740:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203677.188748:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203677/real 1788203677] req@ffff8e828793c380 x1875062187861376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203677.188755:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187861376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203677.188760:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203677.188762:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203677.188764:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187861376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203677.188779:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187861504:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203677.188789:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203677.188795:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203677/real 1788203677] req@ffff8e828793d880 x1875062187861504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203677.188801:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187861504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203677.188811:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203677.188813:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203677.188816:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187861504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203677.188821:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187861632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203677.188838:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203677.188848:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203677/real 1788203677] req@ffff8e828793fb80 x1875062187861632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203677.188858:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187861632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203677.188865:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203677.188867:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203677.188870:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187861632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203677.188875:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187861760:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203677.188893:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203677.188902:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203677/real 1788203677] req@ffff8e828793d500 x1875062187861760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203677.188909:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187861760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203677.188914:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203677.188916:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203677.188918:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187861760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203677.892487:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203678.916477:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203679.940477:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203680.964490:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203681.988660:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203682.308390:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203682.308401:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203682.308406:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203682.308419:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5137 00000100:00080000:0.0:1788203682.308433:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203682.308444:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203682.308485:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187861888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203682.308514:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203682.308518:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203682.308519:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203682.308523:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5137 00000100:00080000:0.0:1788203682.308526:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203682.308529:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203682.308540:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203682.308557:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187862016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203682.308565:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187861888:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203682.308572:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203682.308577:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203682.308579:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203682.308583:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5137 00000100:00080000:0.0:1788203682.308588:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203682.308591:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203682.308596:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203682.308615:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187862144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203682.308623:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203682.308626:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203682.308627:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203682.308630:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5137 00000100:00080000:0.0:1788203682.308633:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788203682.308637:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203682.308640:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203682.308645:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203682.308653:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187862272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203682.308659:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203682.308662:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203682.308663:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203682.308666:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5137 00000100:00080000:0.0:1788203682.308669:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203682.308671:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203682.308674:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203682.308678:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203682/real 1788203682] req@ffff8e828793f480 x1875062187861888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203682.308683:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187862400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203682.308697:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187861888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203682.308721:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203682.308726:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203682.308731:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187861888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203682.308761:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187862016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203682.308781:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203682.308789:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203682/real 1788203682] req@ffff8e828793dc00 x1875062187862016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203682.308797:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187862016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203682.308802:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203682.308804:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203682.308806:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187862016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203682.308809:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187862144:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203682.308827:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203682.308839:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203682/real 1788203682] req@ffff8e828793c000 x1875062187862144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203682.308851:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187862144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203682.308859:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203682.308862:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203682.308865:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187862144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203682.308871:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187862272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203682.308891:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203682.308899:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203682/real 1788203682] req@ffff8e828793ca80 x1875062187862272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203682.308911:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187862272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203682.308916:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203682.308919:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203682.308922:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187862272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203682.308929:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187862400:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203682.308945:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203682.308953:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203682/real 1788203682] req@ffff8e828793c700 x1875062187862400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203682.308962:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187862400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203682.308980:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203682.308983:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203682.308985:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187862400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203683.012458:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203684.036534:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203685.060513:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203686.084509:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203687.108539:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203687.428401:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203687.428409:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203687.428412:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203687.428420:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5142 00000100:00080000:0.0:1788203687.428428:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203687.428439:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203687.428467:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187862528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203687.428487:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203687.428490:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203687.428491:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203687.428494:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5142 00000100:00080000:0.0:1788203687.428496:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203687.428498:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203687.428504:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203687.428512:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187862656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203687.428519:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203687.428521:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203687.428522:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203687.428525:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5142 00000100:00080000:0.0:1788203687.428528:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788203687.428529:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187862528:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203687.428533:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203687.428538:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203687.428544:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187862784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203687.428549:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203687.428551:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203687.428552:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203687.428555:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5142 00000100:00080000:0.0:1788203687.428556:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203687.428558:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203687.428561:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203687.428568:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187862912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203687.428573:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203687.428575:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203687.428575:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203687.428577:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5142 00000100:00080000:0.0:1788203687.428587:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203687.428589:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203687.428591:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788203687.428598:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203687.428601:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187863040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203687.428640:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203687/real 1788203687] req@ffff8e828793f100 x1875062187862528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203687.428657:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187862528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203687.428664:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203687.428666:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203687.428669:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187862528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203687.428685:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187862656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203687.428700:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203687.428708:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203687/real 1788203687] req@ffff8e828793e680 x1875062187862656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203687.428714:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187862656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203687.428719:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203687.428721:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203687.428723:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187862656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203687.428726:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187862784:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203687.428734:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203687.428739:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203687/real 1788203687] req@ffff8e828793f800 x1875062187862784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203687.428745:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187862784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203687.428753:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203687.428755:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203687.428757:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187862784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203687.428760:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187862912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203687.428768:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203687.428773:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203687/real 1788203687] req@ffff8e828793ce00 x1875062187862912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203687.428779:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187862912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203687.428783:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203687.428785:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203687.428786:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187862912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203687.428789:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187863040:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203687.428797:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203687.428802:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203687/real 1788203687] req@ffff8e828793c700 x1875062187863040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203687.428808:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187863040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203687.428812:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203687.428813:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203687.428815:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187863040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203688.132527:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203689.156485:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203690.180461:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203691.204465:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203692.228465:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203692.548449:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203692.548459:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203692.548463:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203692.548472:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5147 00000100:00080000:3.0:1788203692.548481:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203692.548486:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203692.548515:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187863168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203692.548536:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203692.548540:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203692.548542:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203692.548544:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5147 00000100:00080000:3.0:1788203692.548547:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203692.548550:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203692.548559:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203692.548571:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187863168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203692.548574:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187863296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203692.548587:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203692.548591:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203692.548593:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203692.548597:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5147 00000100:00080000:3.0:1788203692.548600:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203692.548603:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203692.548608:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203692.548621:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187863424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203692.548623:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788203692.548634:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203692.548638:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203692.548640:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203692.548644:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5147 00000100:00080000:3.0:1788203692.548648:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203692.548652:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203692.548657:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203692/real 1788203692] req@ffff8e8289538a80 x1875062187863168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788203692.548659:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203692.548679:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187863168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203692.548679:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187863552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203692.548697:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203692.548700:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203692.548705:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187863168:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788203692.548711:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203692.548715:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203692.548716:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203692.548720:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5147 00000100:00080000:3.0:1788203692.548723:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203692.548725:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203692.548730:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203692.548732:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187863296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203692.548740:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187863680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203692.548759:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203692.548773:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203692/real 1788203692] req@ffff8e828953a680 x1875062187863296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203692.548785:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187863296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203692.548818:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203692.548821:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203692.548824:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187863296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203692.548830:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187863424:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203692.548848:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203692.548859:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203692/real 1788203692] req@ffff8e828953b100 x1875062187863424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203692.548870:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187863424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203692.548877:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203692.548880:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203692.548883:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187863424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203692.548889:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187863552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203692.548921:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203692.548930:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203692/real 1788203692] req@ffff8e8289539500 x1875062187863552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203692.548937:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187863552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203692.548942:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203692.548944:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203692.548945:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187863552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203692.548955:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187863680:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203692.548967:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203692.548973:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203692/real 1788203692] req@ffff8e8289539880 x1875062187863680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203692.548980:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187863680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203692.548984:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203692.548986:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203692.548987:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187863680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203693.252452:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203694.276449:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203695.300501:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203696.324506:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203697.348468:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203697.668440:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203697.668450:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203697.668454:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203697.668464:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5152 00000100:00080000:2.0:1788203697.668472:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203697.668480:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203697.668514:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187863808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203697.668541:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203697.668545:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203697.668547:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203697.668551:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5152 00000100:00080000:2.0:1788203697.668555:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203697.668559:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203697.668568:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203697.668581:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187863936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203697.668592:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788203697.668596:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187863808:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203697.668596:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203697.668599:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203697.668607:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5152 00000100:00080000:2.0:1788203697.668611:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203697.668615:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203697.668621:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203697.668633:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187864064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203697.668641:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203697.668643:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203697.668645:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203697.668647:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5152 00000100:00080000:2.0:1788203697.668650:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203697.668653:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203697.668656:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788203697.668661:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203697.668665:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187864192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203697.668672:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203697.668674:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203697.668675:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203697.668678:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5152 00000100:00080000:2.0:1788203697.668682:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203697.668685:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203697.668689:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203697.668713:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187864320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203697.668722:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203697/real 1788203697] req@ffff8e82894e6d80 x1875062187863808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203697.668744:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187863808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203697.668754:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203697.668758:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203697.668761:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187863808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203697.668785:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187863936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203697.668805:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203697.668816:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203697/real 1788203697] req@ffff8e82894e5f80 x1875062187863936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203697.668831:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187863936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203697.668839:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203697.668842:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203697.668845:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187863936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203697.668850:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187864064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203697.668865:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203697.668873:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203697/real 1788203697] req@ffff8e82894e4380 x1875062187864064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203697.668883:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187864064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203697.668890:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203697.668893:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203697.668896:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187864064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203697.668901:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187864192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203697.668915:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203697.668923:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203697/real 1788203697] req@ffff8e82894e5c00 x1875062187864192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203697.668933:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187864192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203697.668945:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203697.668948:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203697.668951:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187864192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203697.668956:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187864320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203697.668972:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203697.668981:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203697/real 1788203697] req@ffff8e82894e7100 x1875062187864320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203697.668999:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187864320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203697.669006:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203697.669009:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203697.669011:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187864320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203698.372502:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203699.396470:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203700.420450:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203701.444469:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203702.468493:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203702.788465:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203702.788480:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203702.788485:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203702.788500:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5157 00000100:00080000:2.0:1788203702.788522:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203702.788535:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203702.788588:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187864448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203702.788635:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203702.788639:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203702.788641:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203702.788646:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5157 00000100:00080000:2.0:1788203702.788649:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203702.788654:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203702.788664:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203702.788678:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187864576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203702.788688:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203702.788692:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203702.788694:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203702.788697:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5157 00000100:00080000:2.0:1788203702.788700:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203702.788704:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203702.788709:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203702.788718:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187864704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203702.788728:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203702.788732:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203702.788734:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203702.788737:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5157 00000100:00080000:2.0:1788203702.788740:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203702.788743:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203702.788747:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203702.788755:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187864832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203702.788764:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203702.788768:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203702.788770:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203702.788774:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5157 00000100:00080000:2.0:1788203702.788777:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203702.788781:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203702.788787:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187864448:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788203702.788788:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203702.788801:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187864960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203702.788916:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187864576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203702.788953:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187864704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203702.788978:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187864832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203702.789016:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187864960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203703.492552:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203704.516441:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203705.540512:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203705.860704:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788203705.860758:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788203705.860790:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203705.860841:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203705.860855:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203705.860864:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203705.860873:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788203705.861025:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203702/real 1788203705] req@ffff8e82894e4000 x1875062187864960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203705.861054:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187864960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203705.861068:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203705.861073:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203705.861078:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187864960:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203705.861086:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203702/real 1788203705] req@ffff8e82894e6300 x1875062187864448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203705.861100:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187864448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203705.861108:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203705.861111:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203705.861115:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187864448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203705.861122:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203702/real 1788203705] req@ffff8e82894e7800 x1875062187864576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203705.861133:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187864576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203705.861149:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203705.861153:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203705.861156:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187864576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203705.861163:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203702/real 1788203705] req@ffff8e82894e4a80 x1875062187864704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203705.861173:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187864704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203705.861179:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203705.861182:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203705.861185:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187864704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203705.861191:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203702/real 1788203705] req@ffff8e82894e4e00 x1875062187864832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203705.861202:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187864832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203705.861209:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203705.861212:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203705.861215:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187864832:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203706.564516:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203707.588497:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203707.908488:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203707.908499:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203707.908503:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203707.908514:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5162 00000100:00080000:3.0:1788203707.908538:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203707.908548:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203707.908592:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187865088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203707.908617:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203707.908621:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203707.908623:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203707.908627:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5162 00000100:00080000:3.0:1788203707.908631:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203707.908636:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203707.908648:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203707.908661:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187865216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203707.908667:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187865088:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788203707.908671:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203707.908674:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203707.908676:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203707.908680:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5162 00000100:00080000:3.0:1788203707.908683:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203707.908686:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203707.908696:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203707.908706:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187865344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203707.908713:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203707.908716:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203707.908718:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203707.908721:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5162 00000100:00080000:3.0:1788203707.908724:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203707.908727:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203707.908732:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203707.908742:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187865472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203707.908748:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788203707.908750:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203707.908753:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203707.908755:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203707.908758:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5162 00000100:00080000:3.0:1788203707.908760:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203707.908764:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203707.908768:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203707.908777:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187865600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203707.908799:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203707/real 1788203707] req@ffff8e828953bb80 x1875062187865088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203707.908824:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187865088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203707.908835:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203707.908840:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203707.908844:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187865088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203707.908875:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187865216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203707.908899:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203707.908912:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203707/real 1788203707] req@ffff8e8289538380 x1875062187865216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203707.908924:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187865216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203707.908931:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203707.908934:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203707.908938:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187865216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203707.908943:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187865344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203707.908965:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203707.908977:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203707/real 1788203707] req@ffff8e8289539f80 x1875062187865344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203707.908993:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187865344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203707.909001:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203707.909003:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203707.909006:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187865344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203707.909012:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187865472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203707.909031:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203707.909041:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203707/real 1788203707] req@ffff8e8289539880 x1875062187865472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203707.909051:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187865472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203707.909057:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203707.909060:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203707.909063:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187865472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203707.909068:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187865600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203707.909084:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203707.909095:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203707/real 1788203707] req@ffff8e8289539500 x1875062187865600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203707.909104:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187865600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203707.909118:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203707.909121:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203707.909124:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187865600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203708.612595:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203709.636442:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203710.660438:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203711.684476:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203712.708462:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203713.028442:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203713.028454:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203713.028459:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203713.028470:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5168 00000100:00080000:1.0:1788203713.028483:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203713.028491:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203713.028537:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187865728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203713.028565:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203713.028567:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203713.028569:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203713.028572:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5168 00000100:00080000:1.0:1788203713.028574:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203713.028577:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203713.028586:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203713.028594:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187865856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203713.028601:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203713.028604:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203713.028605:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203713.028607:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5168 00000100:00080000:1.0:1788203713.028610:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203713.028613:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203713.028618:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203713.028625:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187865728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203713.028630:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187865984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203713.028641:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203713.028645:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203713.028647:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203713.028650:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5168 00000100:00080000:1.0:1788203713.028653:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203713.028657:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203713.028661:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203713.028669:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187866112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203713.028675:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203713.028678:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203713.028679:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203713.028681:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5168 00000100:00080000:1.0:1788203713.028684:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203713.028686:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203713.028689:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203713.028696:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187866240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203713.028733:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203713.028813:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203713/real 1788203713] req@ffff8e828d722300 x1875062187865728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203713.028838:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187865728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203713.028849:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203713.028853:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203713.028857:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187865728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203713.028866:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187865856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203713.028890:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203713.028902:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203713/real 1788203713] req@ffff8e828d722d80 x1875062187865856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203713.028910:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187865856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203713.028916:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203713.028918:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203713.028920:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187865856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203713.028943:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187865984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203713.028956:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203713.028962:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203713/real 1788203713] req@ffff8e828d721180 x1875062187865984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203713.028969:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187865984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203713.028978:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203713.028980:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203713.028982:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187865984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203713.028985:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187866112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203713.028995:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203713.029000:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203713/real 1788203713] req@ffff8e828d720380 x1875062187866112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203713.029007:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187866112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203713.029011:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203713.029013:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203713.029015:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187866112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203713.029017:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187866240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203713.029030:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203713.029038:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203713/real 1788203713] req@ffff8e828d720a80 x1875062187866240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203713.029048:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187866240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203713.029056:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203713.029059:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203713.029063:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187866240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203713.732505:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203714.756543:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203715.780536:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203716.804506:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203717.828540:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203718.148520:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203718.148530:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203718.148536:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203718.148549:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5173 00000100:00080000:1.0:1788203718.148562:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203718.148572:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203718.148616:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187866368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203718.148648:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203718.148651:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203718.148653:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203718.148656:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5173 00000100:00080000:1.0:1788203718.148660:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203718.148663:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203718.148675:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203718.148691:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187866368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203718.148692:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187866496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203718.148713:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203718.148717:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203718.148719:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203718.148723:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5173 00000100:00080000:1.0:1788203718.148727:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203718.148730:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203718.148735:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203718.148745:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187866624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203718.148755:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203718.148758:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203718.148759:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203718.148763:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5173 00000100:00080000:1.0:1788203718.148765:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203718.148767:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203718.148771:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203718.148780:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187866752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203718.148780:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788203718.148788:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203718.148791:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203718.148792:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203718.148794:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5173 00000100:00080000:1.0:1788203718.148797:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203718.148800:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203718.148803:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203718.148808:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187866880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203718.149027:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203718/real 1788203718] req@ffff8e828d720a80 x1875062187866368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203718.149044:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187866368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203718.149056:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203718.149097:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203718.149102:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187866368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203718.149128:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187866496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203718.149146:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203718.149156:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203718/real 1788203718] req@ffff8e828d720380 x1875062187866496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203718.149163:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187866496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203718.149168:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203718.149170:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203718.149172:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187866496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203718.149176:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187866624:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203718.149186:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203718.149191:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203718/real 1788203718] req@ffff8e828d721180 x1875062187866624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203718.149197:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187866624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203718.149202:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203718.149203:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203718.149205:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187866624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203718.149208:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187866752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203718.149218:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203718.149223:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203718/real 1788203718] req@ffff8e828d722d80 x1875062187866752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203718.149230:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187866752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203718.149234:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203718.149236:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203718.149237:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187866752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203718.149240:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187866880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203718.149249:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203718.149253:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203718/real 1788203718] req@ffff8e828d722300 x1875062187866880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203718.149260:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187866880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203718.149264:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203718.149265:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203718.149267:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187866880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203718.852496:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203719.876528:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203720.900482:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203721.924433:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203722.948505:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203723.268436:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203723.268447:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203723.268452:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203723.268465:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5178 00000100:00080000:0.0:1788203723.268480:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203723.268488:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203723.268542:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187867008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203723.268566:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203723.268569:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203723.268570:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203723.268573:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5178 00000100:00080000:0.0:1788203723.268576:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203723.268579:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203723.268589:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203723.268598:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187867136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203723.268606:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187867008:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203723.268609:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203723.268614:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203723.268616:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203723.268621:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5178 00000100:00080000:0.0:1788203723.268624:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203723.268629:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203723.268634:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203723.268649:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187867264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203723.268659:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203723.268662:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203723.268664:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203723.268668:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5178 00000100:00080000:0.0:1788203723.268671:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788203723.268676:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203723.268677:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203723.268685:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203723.268698:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187867392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203723.268706:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203723.268708:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203723.268710:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203723.268713:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5178 00000100:00080000:0.0:1788203723.268716:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203723.268719:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203723.268722:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203723/real 1788203723] req@ffff8e828793ca80 x1875062187867008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788203723.268724:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203723.268738:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187867520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203723.268754:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187867008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203723.268763:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203723.268767:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203723.268770:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187867008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203723.268794:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187867136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203723.268816:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203723.268827:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203723/real 1788203723] req@ffff8e828793c000 x1875062187867136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203723.268846:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187867136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203723.268851:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203723.268853:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203723.268856:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187867136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203723.268859:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187867264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203723.268873:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203723.268880:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203723/real 1788203723] req@ffff8e828793dc00 x1875062187867264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203723.268888:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187867264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203723.268897:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203723.268899:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203723.268901:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187867264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203723.268904:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187867392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203723.268913:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203723.268918:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203723/real 1788203723] req@ffff8e828793f480 x1875062187867392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203723.268925:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187867392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203723.268929:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203723.268931:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203723.268932:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187867392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203723.268935:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187867520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203723.268944:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203723.268949:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203723/real 1788203723] req@ffff8e828793d500 x1875062187867520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203723.268954:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187867520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203723.268959:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203723.268961:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203723.268963:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187867520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203723.972496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203724.996511:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203726.020489:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203727.044489:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203728.068483:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203728.388394:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203728.388408:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203728.388410:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203728.388418:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5183 00000100:00080000:1.0:1788203728.388426:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203728.388431:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203728.388460:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187867648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203728.388477:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203728.388480:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203728.388481:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203728.388487:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5183 00000100:00080000:1.0:1788203728.388490:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203728.388492:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203728.388501:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203728.388509:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187867776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203728.388515:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187867648:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788203728.388518:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203728.388524:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203728.388525:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203728.388528:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5183 00000100:00080000:1.0:1788203728.388530:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203728.388532:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203728.388534:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203728.388541:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187867904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203728.388546:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203728.388548:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203728.388550:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203728.388552:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5183 00000100:00080000:1.0:1788203728.388554:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203728.388556:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203728.388559:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203728.388564:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187868032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203728.388569:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203728.388571:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203728.388572:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203728.388574:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5183 00000100:00080000:1.0:1788203728.388576:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203728.388580:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:3.0:1788203728.388583:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:1.0:1788203728.388584:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203728.388594:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187868160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203728.388638:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203728/real 1788203728] req@ffff8e828d720e00 x1875062187867648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203728.388656:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187867648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203728.388666:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203728.388670:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203728.388673:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187867648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203728.388693:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187867776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203728.388712:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203728.388722:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203728/real 1788203728] req@ffff8e828d723800 x1875062187867776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203728.388732:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187867776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203728.388739:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203728.388742:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203728.388745:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187867776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203728.388750:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187867904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203728.388764:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203728.388773:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203728/real 1788203728] req@ffff8e828d721500 x1875062187867904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203728.388784:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187867904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203728.388792:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203728.388794:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203728.388798:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187867904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203728.388802:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187868032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203728.388816:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203728.388825:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203728/real 1788203728] req@ffff8e828d720000 x1875062187868032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203728.388836:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187868032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203728.388842:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203728.388845:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203728.388847:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187868032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203728.388852:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187868160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203728.388869:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203728.388877:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203728/real 1788203728] req@ffff8e828d722680 x1875062187868160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203728.388888:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187868160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203728.388895:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203728.388898:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203728.388900:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187868160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203729.092473:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203730.116509:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203731.140487:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203732.164491:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203733.188523:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203733.508441:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203733.508450:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203733.508453:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203733.508462:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5188 00000100:00080000:1.0:1788203733.508471:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203733.508477:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203733.508512:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187868288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203733.508536:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203733.508539:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203733.508541:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203733.508544:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5188 00000100:00080000:1.0:1788203733.508547:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203733.508549:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203733.508556:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203733.508566:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187868416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203733.508574:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203733.508576:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203733.508577:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203733.508580:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5188 00000100:00080000:1.0:1788203733.508582:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203733.508584:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203733.508588:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203733.508600:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187868288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203733.508605:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187868544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203733.508616:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203733.508618:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203733.508619:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203733.508621:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5188 00000100:00080000:1.0:1788203733.508623:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203733.508626:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203733.508631:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203733.508643:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187868672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203733.508653:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203733.508656:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203733.508658:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788203733.508663:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788203733.508677:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5188 00000100:00080000:1.0:1788203733.508684:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203733.508688:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203733.508695:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203733.508705:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187868800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203733.508717:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203733/real 1788203733] req@ffff8e828d722680 x1875062187868288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203733.508738:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187868288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203733.508747:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203733.508751:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203733.508754:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187868288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203733.508759:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187868416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203733.508778:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203733.508787:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203733/real 1788203733] req@ffff8e828d720000 x1875062187868416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203733.508795:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187868416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203733.508801:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203733.508803:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203733.508805:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187868416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203733.508823:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187868544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203733.508834:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203733.508842:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203733/real 1788203733] req@ffff8e828d721500 x1875062187868544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203733.508850:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187868544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203733.508855:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203733.508857:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203733.508859:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187868544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203733.508863:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187868672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203733.508873:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203733.508879:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203733/real 1788203733] req@ffff8e828d723800 x1875062187868672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203733.508887:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187868672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203733.508892:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203733.508893:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203733.508895:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187868672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203733.508898:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187868800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203733.508907:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203733.508913:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203733/real 1788203733] req@ffff8e828d720e00 x1875062187868800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203733.508919:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187868800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203733.508925:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203733.508927:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203733.508929:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187868800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203734.212521:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203735.236488:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203736.260445:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203737.284478:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203738.308501:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203738.628414:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203738.628423:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203738.628426:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203738.628435:0:20954:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5193 00000100:00080000:3.0:1788203738.628444:0:20954:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203738.628449:0:20954:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203738.628480:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187868928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203738.628500:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203738.628503:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203738.628504:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203738.628507:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5193 00000100:00080000:3.0:1788203738.628509:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203738.628513:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203738.628521:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203738.628534:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187868928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203738.628534:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187869056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203738.628550:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203738.628552:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203738.628554:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203738.628556:0:20954:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5193 00000100:00080000:3.0:1788203738.628558:0:20954:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203738.628561:0:20954:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203738.628564:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203738.628570:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187869184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203738.628577:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203738.628580:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203738.628581:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203738.628583:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5193 00000100:00080000:3.0:1788203738.628593:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203738.628596:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203738.628600:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788203738.628611:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203738.628613:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187869312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203738.628625:0:20954:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203738.628629:0:20954:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203738.628631:0:20954:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203738.628636:0:20954:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5193 00000100:00080000:3.0:1788203738.628640:0:20954:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203738.628644:0:20954:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203738.628650:0:20954:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203738.628664:0:20954:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187869440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203738.628672:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203738/real 1788203738] req@ffff8e828953b100 x1875062187868928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203738.628696:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187868928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203738.628708:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203738.628713:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203738.628717:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187868928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203738.628749:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187869056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203738.628780:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203738.628795:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203738/real 1788203738] req@ffff8e828953a680 x1875062187869056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203738.628808:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187869056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203738.628816:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203738.628820:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203738.628823:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187869056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203738.628830:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187869184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203738.628848:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203738.628859:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203738/real 1788203738] req@ffff8e8289538a80 x1875062187869184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203738.628870:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187869184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203738.628878:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203738.628882:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203738.628885:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187869184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203738.628891:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187869312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203738.628909:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203738.628921:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203738/real 1788203738] req@ffff8e8289539500 x1875062187869312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203738.628933:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187869312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203738.628942:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203738.628945:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203738.628948:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187869312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203738.628953:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187869440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203738.628974:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203738.628986:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203738/real 1788203738] req@ffff8e8289539880 x1875062187869440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203738.629029:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187869440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203738.629038:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203738.629041:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203738.629044:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187869440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203739.332470:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203740.356490:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203741.380476:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203742.404458:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203743.428490:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203743.748447:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203743.748462:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203743.748468:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203743.748482:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5198 00000100:00080000:0.0:1788203743.748497:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203743.748506:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203743.748560:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187869568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203743.748598:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203743.748603:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203743.748605:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203743.748611:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5198 00000100:00080000:0.0:1788203743.748615:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203743.748620:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203743.748631:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203743.748658:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187869696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203743.748671:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203743.748674:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203743.748676:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203743.748681:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5198 00000100:00080000:0.0:1788203743.748685:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203743.748689:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203743.748695:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203743.748706:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187869568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203743.748715:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187869824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203743.748729:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203743.748733:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203743.748735:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203743.748740:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5198 00000100:00080000:0.0:1788203743.748744:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203743.748748:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203743.748755:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203743.748769:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187869952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203743.748779:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203743.748782:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203743.748788:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203743.748790:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203743.748799:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5198 00000100:00080000:0.0:1788203743.748804:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203743.748810:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203743.748818:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203743.748838:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203743/real 1788203743] req@ffff8e828793fb80 x1875062187869568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203743.748841:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187870080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203743.748889:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187869568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203743.748917:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203743.748922:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203743.748926:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187869568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203743.748934:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187869696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203743.748955:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203743.748965:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203743/real 1788203743] req@ffff8e828793d880 x1875062187869696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203743.748974:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187869696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203743.748985:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203743.748988:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203743.748992:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187869696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203743.749018:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187869824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203743.749042:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203743.749055:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203743/real 1788203743] req@ffff8e828793c380 x1875062187869824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203743.749064:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187869824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203743.749070:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203743.749073:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203743.749075:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187869824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203743.749080:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187869952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203743.749095:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203743.749103:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203743/real 1788203743] req@ffff8e828793e300 x1875062187869952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203743.749110:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187869952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203743.749115:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203743.749117:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203743.749129:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187869952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203743.749141:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187870080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203743.749162:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203743.749175:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203743/real 1788203743] req@ffff8e828793d500 x1875062187870080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203743.749188:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187870080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203743.749195:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203743.749197:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203743.749199:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187870080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203744.452547:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203745.476557:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203746.500467:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203747.524549:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203748.548482:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203748.868453:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203748.868464:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203748.868469:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203748.868487:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5203 00000100:00080000:2.0:1788203748.868499:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203748.868507:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203748.868553:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187870208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203748.868590:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203748.868595:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203748.868597:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203748.868603:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5203 00000100:00080000:2.0:1788203748.868608:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203748.868614:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203748.868627:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203748.868645:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187870208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203748.868647:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187870336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203748.868663:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203748.868667:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203748.868670:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203748.868674:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5203 00000100:00080000:2.0:1788203748.868679:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203748.868684:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203748.868691:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203748.868706:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187870464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203748.868714:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203748.868717:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203748.868722:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788203748.868722:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203748.868729:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5203 00000100:00080000:2.0:1788203748.868734:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203748.868737:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203748.868742:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203748.868752:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187870592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203748.868759:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203748.868762:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203748.868764:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203748.868767:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5203 00000100:00080000:2.0:1788203748.868770:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203748.868772:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203748.868776:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203748.868783:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203748/real 1788203748] req@ffff8e82894e7480 x1875062187870208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203748.868788:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187870720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203748.868841:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187870208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203748.868851:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203748.868854:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203748.868857:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187870208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203748.868884:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187870336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203748.868903:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203748.868913:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203748/real 1788203748] req@ffff8e82894e6a00 x1875062187870336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203748.868923:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187870336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203748.868929:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203748.868931:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203748.868933:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187870336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203748.868937:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187870464:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203748.868951:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203748.868958:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203748/real 1788203748] req@ffff8e82894e5880 x1875062187870464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203748.868966:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187870464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203748.868971:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203748.868973:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203748.868975:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187870464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203748.868978:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187870592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203748.868990:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203748.868996:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203748/real 1788203748] req@ffff8e82894e6680 x1875062187870592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203748.869003:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187870592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203748.869008:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203748.869010:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203748.869012:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187870592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203748.869015:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187870720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203748.869025:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203748.869030:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203748/real 1788203748] req@ffff8e82894e4e00 x1875062187870720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203748.869051:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187870720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203748.869057:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203748.869059:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203748.869061:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187870720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203749.572546:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203750.596513:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203751.620515:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203752.644466:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203753.668489:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203753.988497:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203753.988505:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203753.988509:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203753.988521:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5209 00000100:00080000:0.0:1788203753.988532:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203753.988541:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203753.988580:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187870848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203753.988602:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203753.988606:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203753.988607:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203753.988611:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5209 00000100:00080000:0.0:1788203753.988615:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203753.988618:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203753.988627:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203753.988635:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187870848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203753.988640:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187870976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203753.988649:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203753.988652:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203753.988654:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203753.988656:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5209 00000100:00080000:0.0:1788203753.988660:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203753.988663:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203753.988667:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203753.988677:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187871104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203753.988685:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203753.988688:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203753.988689:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203753.988692:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5209 00000100:00080000:0.0:1788203753.988696:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203753.988700:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203753.988703:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788203753.988708:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203753.988714:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187871232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203753.988733:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203753.988736:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203753.988738:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203753.988741:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5209 00000100:00080000:0.0:1788203753.988744:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203753.988747:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203753.988751:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203753.988760:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203753/real 1788203753] req@ffff8e828793f480 x1875062187870848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203753.988763:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187871360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203753.988794:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187870848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203753.988803:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203753.988807:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203753.988810:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187870848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203753.988830:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187870976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203753.988853:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203753.988867:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203753/real 1788203753] req@ffff8e828793dc00 x1875062187870976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203753.988881:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187870976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203753.988891:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203753.988895:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203753.988899:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187870976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203753.988907:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187871104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203753.988926:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203753.988934:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203753/real 1788203753] req@ffff8e828793c000 x1875062187871104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203753.988942:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187871104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203753.988952:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203753.988956:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203753.988959:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187871104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203753.988966:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187871232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203753.988993:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203753.989009:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203753/real 1788203753] req@ffff8e828793ca80 x1875062187871232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203753.989022:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187871232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203753.989029:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203753.989031:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203753.989035:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187871232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203753.989041:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187871360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203753.989060:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203753.989075:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203753/real 1788203753] req@ffff8e828793c700 x1875062187871360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203753.989088:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187871360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203753.989096:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203753.989099:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203753.989102:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187871360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203754.692501:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203755.716498:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203756.740476:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203757.764490:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203758.788510:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203759.108410:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203759.108419:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203759.108423:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203759.108431:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5214 00000100:00080000:0.0:1788203759.108440:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203759.108445:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203759.108475:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187871488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203759.108498:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203759.108501:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203759.108503:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203759.108506:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5214 00000100:00080000:0.0:1788203759.108508:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203759.108510:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203759.108517:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203759.108526:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187871616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203759.108533:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203759.108536:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203759.108537:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203759.108540:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5214 00000100:00080000:0.0:1788203759.108541:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203759.108544:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203759.108547:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203759.108555:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187871744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203759.108559:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187871488:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203759.108566:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203759.108570:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203759.108571:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203759.108573:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5214 00000100:00080000:0.0:1788203759.108575:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203759.108577:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203759.108581:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203759.108588:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187871872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203759.108594:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203759.108596:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203759.108597:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203759.108600:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5214 00000100:00080000:0.0:1788203759.108604:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203759.108607:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203759.108610:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203759.108616:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187872000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203759.108656:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203759.108693:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203759/real 1788203759] req@ffff8e828793ce00 x1875062187871488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203759.108707:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187871488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203759.108715:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203759.108718:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203759.108721:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187871488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203759.108726:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187871616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203759.108743:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203759.108752:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203759/real 1788203759] req@ffff8e828793f800 x1875062187871616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203759.108760:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187871616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203759.108765:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203759.108768:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203759.108770:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187871616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203759.108788:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187871744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203759.108799:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203759.108805:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203759/real 1788203759] req@ffff8e828793e680 x1875062187871744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203759.108815:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187871744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203759.108823:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203759.108827:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203759.108830:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187871744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203759.108836:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187871872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203759.108858:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203759.108871:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203759/real 1788203759] req@ffff8e828793f100 x1875062187871872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203759.108883:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187871872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203759.108892:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203759.108895:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203759.108898:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187871872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203759.108902:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187872000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203759.108918:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203759.108925:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203759/real 1788203759] req@ffff8e828793c700 x1875062187872000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203759.108933:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187872000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203759.108948:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203759.108953:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203759.108957:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187872000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203759.812470:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203760.836455:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203761.860481:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203762.884423:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203763.908496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203764.228447:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203764.228457:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203764.228461:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203764.228471:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5219 00000100:00080000:1.0:1788203764.228482:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203764.228489:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203764.228525:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187872128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203764.228564:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203764.228571:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203764.228573:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203764.228578:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5219 00000100:00080000:1.0:1788203764.228582:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203764.228586:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203764.228595:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203764.228609:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187872256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203764.228621:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187872128:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788203764.228624:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203764.228632:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203764.228634:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203764.228639:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5219 00000100:00080000:1.0:1788203764.228642:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203764.228646:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203764.228650:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203764.228663:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187872384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203764.228672:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203764.228676:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203764.228679:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203764.228682:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5219 00000100:00080000:1.0:1788203764.228686:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203764.228689:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:2.0:1788203764.228695:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:1.0:1788203764.228698:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203764.228710:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187872512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203764.228720:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203764.228724:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203764.228725:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203764.228729:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5219 00000100:00080000:1.0:1788203764.228733:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203764.228737:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203764.228741:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203764.228746:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203764/real 1788203764] req@ffff8e828d722300 x1875062187872128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203764.228755:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187872640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203764.228775:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187872128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203764.228789:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203764.228794:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203764.228798:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187872128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203764.228822:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187872256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203764.228846:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203764.228858:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203764/real 1788203764] req@ffff8e828d722d80 x1875062187872256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203764.228871:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187872256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203764.228880:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203764.228884:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203764.228887:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187872256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203764.228894:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187872384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203764.228910:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203764.228920:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203764/real 1788203764] req@ffff8e828d721180 x1875062187872384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203764.228932:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187872384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203764.228940:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203764.228943:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203764.228946:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187872384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203764.228952:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187872512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203764.228969:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203764.228981:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203764/real 1788203764] req@ffff8e828d720380 x1875062187872512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203764.228995:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187872512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203764.229003:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203764.229007:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203764.229010:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187872512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203764.229017:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187872640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203764.229039:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203764.229052:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203764/real 1788203764] req@ffff8e828d720a80 x1875062187872640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203764.229064:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187872640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203764.229073:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203764.229077:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203764.229080:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187872640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203764.932514:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203765.956512:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203766.980479:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203768.004508:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203769.028511:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203769.348441:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203769.348452:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203769.348467:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203769.348478:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5224 00000100:00080000:3.0:1788203769.348489:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203769.348496:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203769.348528:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187872768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203769.348553:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203769.348556:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203769.348557:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203769.348560:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5224 00000100:00080000:3.0:1788203769.348563:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203769.348565:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203769.348579:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203769.348597:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187872896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203769.348616:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788203769.348620:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187872768:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788203769.348621:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203769.348626:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203769.348642:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5224 00000100:00080000:3.0:1788203769.348646:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203769.348649:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203769.348654:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203769.348666:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187873024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203769.348673:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203769.348676:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203769.348677:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203769.348681:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5224 00000100:00080000:3.0:1788203769.348684:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203769.348686:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203769.348690:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203769.348699:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187873152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203769.348705:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203769.348708:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203769.348709:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203769.348711:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5224 00000100:00080000:3.0:1788203769.348714:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203769.348716:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203769.348720:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203769.348727:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187873280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203769.348757:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187872896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203769.348796:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187873024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203769.348824:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187873152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203769.348859:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187873280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203770.052500:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203771.076512:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203772.100523:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788203772.420601:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:0.0:1788203772.420650:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788203772.420669:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:0.0:1788203772.420724:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:0.0:1788203772.420734:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:0.0:1788203772.420745:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:0.0:1788203772.420756:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:2.0:1788203772.420808:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203769/real 1788203772] req@ffff8e8289538380 x1875062187872896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203772.420836:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187872896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203772.420848:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203772.420853:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203772.420858:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187872896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203772.420968:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203769/real 1788203772] req@ffff8e828953bb80 x1875062187873024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203772.420985:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187873024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203772.420993:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203772.420997:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203772.421001:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187873024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203772.421009:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203769/real 1788203772] req@ffff8e8289539880 x1875062187873152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203772.421021:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187873152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203772.421029:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203772.421032:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203772.421035:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187873152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203772.421043:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203769/real 1788203772] req@ffff8e8289539500 x1875062187873280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203772.421053:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187873280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203772.421061:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203772.421065:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203772.421068:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187873280:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203772.421075:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203769/real 1788203772] req@ffff8e8289539f80 x1875062187872768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203772.421086:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187872768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203772.421094:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203772.421098:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203772.421101:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187872768:192.168.201.121@tcp:250:kworker.0 00010000:00010000:0.0:1788203773.124503:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203774.148469:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203774.468441:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203774.468450:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203774.468453:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203774.468461:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5229 00000100:00080000:1.0:1788203774.468470:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203774.468476:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203774.468511:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187873408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203774.468542:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203774.468547:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203774.468549:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203774.468554:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5229 00000100:00080000:1.0:1788203774.468558:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203774.468563:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203774.468583:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203774.468591:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187873408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203774.468601:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187873536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203774.468612:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203774.468615:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203774.468616:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203774.468620:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5229 00000100:00080000:1.0:1788203774.468624:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203774.468628:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203774.468633:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203774.468653:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187873664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203774.468666:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788203774.468670:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788203774.468672:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203774.468675:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203774.468681:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5229 00000100:00080000:1.0:1788203774.468686:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203774.468691:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203774.468698:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203774.468711:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187873792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203774.468719:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203774/real 1788203774] req@ffff8e828d720a80 x1875062187873408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203774.468725:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203774.468734:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203774.468739:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203774.468745:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5229 00000100:00080000:1.0:1788203774.468750:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788203774.468751:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187873408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203774.468759:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203774.468769:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788203774.468772:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203774.468780:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203774.468785:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187873920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788203774.468787:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187873408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203774.468822:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187873536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203774.468845:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203774.468858:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203774/real 1788203774] req@ffff8e828d720380 x1875062187873536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203774.468872:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187873536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203774.468881:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203774.468884:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203774.468888:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187873536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203774.468894:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187873664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203774.468909:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203774.468918:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203774/real 1788203774] req@ffff8e828d721180 x1875062187873664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203774.468929:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187873664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203774.468938:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203774.468941:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203774.468944:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187873664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203774.468949:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187873792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203774.468965:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203774.468974:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203774/real 1788203774] req@ffff8e828d722d80 x1875062187873792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203774.468986:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187873792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203774.468995:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203774.468998:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203774.469001:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187873792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203774.469007:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187873920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203774.469021:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203774.469030:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203774/real 1788203774] req@ffff8e828d722300 x1875062187873920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203774.469041:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187873920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203774.469049:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203774.469052:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203774.469055:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187873920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203775.172522:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203776.196546:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203777.220476:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203778.244450:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203779.268517:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203779.588490:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203779.588499:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203779.588503:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203779.588513:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5234 00000100:00080000:0.0:1788203779.588524:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203779.588533:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203779.588581:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187874048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203779.588622:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203779.588626:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203779.588628:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203779.588633:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5234 00000100:00080000:0.0:1788203779.588637:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203779.588641:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203779.588652:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203779.588669:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187874176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203779.588688:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203779.588693:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203779.588695:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203779.588700:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5234 00000100:00080000:0.0:1788203779.588704:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203779.588709:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203779.588715:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203779.588727:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187874048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203779.588733:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187874304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203779.588753:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203779.588756:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203779.588758:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203779.588763:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5234 00000100:00080000:0.0:1788203779.588767:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203779.588771:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203779.588776:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203779.588791:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187874432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203779.588802:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203779.588806:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203779.588808:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203779.588811:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5234 00000100:00080000:0.0:1788203779.588816:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203779.588820:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203779.588827:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203779.588847:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187874560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203779.588847:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203779.588913:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203779/real 1788203779] req@ffff8e828793ca80 x1875062187874048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203779.588937:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187874048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203779.588947:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203779.588950:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203779.588954:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187874048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203779.588960:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187874176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203779.588981:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203779.588990:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203779/real 1788203779] req@ffff8e828793c000 x1875062187874176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203779.588999:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187874176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203779.589005:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203779.589007:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203779.589010:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187874176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203779.589040:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187874304:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203779.589064:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203779.589076:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203779/real 1788203779] req@ffff8e828793dc00 x1875062187874304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203779.589086:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187874304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203779.589097:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203779.589099:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203779.589101:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187874304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203779.589106:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187874432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203779.589128:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203779.589136:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203779/real 1788203779] req@ffff8e828793f480 x1875062187874432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203779.589144:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187874432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203779.589150:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203779.589153:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203779.589155:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187874432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203779.589158:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187874560:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203779.589170:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203779.589176:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203779/real 1788203779] req@ffff8e828793d500 x1875062187874560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203779.589184:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187874560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203779.589189:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203779.589190:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203779.589193:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187874560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203780.292508:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203781.316605:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203782.340521:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203783.364524:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203784.388534:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203784.708440:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203784.708456:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203784.708462:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203784.708483:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5239 00000100:00080000:1.0:1788203784.708498:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203784.708507:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203784.708563:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187874688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203784.708595:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203784.708600:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203784.708602:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203784.708606:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5239 00000100:00080000:1.0:1788203784.708611:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203784.708615:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203784.708627:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203784.708643:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187874816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203784.708654:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203784.708658:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203784.708659:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203784.708664:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5239 00000100:00080000:1.0:1788203784.708668:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203784.708673:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203784.708681:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203784.708693:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187874688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203784.708700:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187874944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203784.708716:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203784.708719:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203784.708720:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203784.708725:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5239 00000100:00080000:1.0:1788203784.708727:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203784.708732:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203784.708737:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203784.708747:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187875072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203784.708755:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203784.708757:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203784.708759:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203784.708761:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5239 00000100:00080000:1.0:1788203784.708763:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203784.708766:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203784.708769:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203784.708779:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187875200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788203784.708790:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203784.708855:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203784/real 1788203784] req@ffff8e828d720e00 x1875062187874688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203784.708873:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187874688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203784.708882:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203784.708887:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203784.708892:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187874688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203784.708898:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187874816:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203784.708917:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203784.708930:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203784/real 1788203784] req@ffff8e828d723800 x1875062187874816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203784.708943:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187874816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203784.708952:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203784.708955:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203784.708959:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187874816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203784.708993:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187874944:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788203784.709010:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203784.709019:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203784/real 1788203784] req@ffff8e828d721500 x1875062187874944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203784.709027:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187874944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203784.709033:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203784.709035:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203784.709037:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187874944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203784.709041:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187875072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203784.709052:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203784.709058:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203784/real 1788203784] req@ffff8e828d720000 x1875062187875072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203784.709076:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187875072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203784.709085:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203784.709088:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203784.709092:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187875072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788203784.709098:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187875200:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788203784.709121:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203784.709133:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203784/real 1788203784] req@ffff8e828d722680 x1875062187875200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203784.709146:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187875200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203784.709157:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203784.709160:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788203784.709164:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187875200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203785.412489:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203786.436551:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203787.460515:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203788.484534:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203789.508475:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203789.828444:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203789.828459:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203789.828465:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203789.828479:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5244 00000100:00080000:3.0:1788203789.828495:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203789.828515:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203789.828564:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187875328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203789.828601:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203789.828607:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203789.828609:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203789.828616:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5244 00000100:00080000:3.0:1788203789.828622:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203789.828628:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203789.828641:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187875328:192.168.201.121@tcp:250:kworker.0 00000008:00100000:3.0:1788203789.828641:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203789.828664:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187875456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203789.828690:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203789.828698:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203789.828700:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203789.828705:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5244 00000100:00080000:3.0:1788203789.828715:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203789.828721:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203789.828728:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788203789.828733:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203789.828747:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187875584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203789.828761:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203789.828765:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203789.828767:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788203789.828790:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203789/real 1788203789] req@ffff8e8289538a80 x1875062187875328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203789.828810:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187875328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203789.828820:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203789.828824:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203789.828828:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187875328:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788203789.828845:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5244 00000100:00080000:3.0:1788203789.828852:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788203789.828855:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187875456:192.168.201.121@tcp:38:kworker.0 00000100:00080000:3.0:1788203789.828860:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:1.0:1788203789.828880:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203789.828889:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203789/real 1788203789] req@ffff8e828953a680 x1875062187875456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203789.828900:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187875456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203789.828906:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203789.828909:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203789.828911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187875456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203789.828915:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187875584:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203789.828928:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203789.828934:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203789/real 1788203789] req@ffff8e828953b100 x1875062187875584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203789.828942:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187875584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203789.828948:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203789.828950:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203789.828953:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187875584:192.168.201.121@tcp:38:kworker.0 00000008:00100000:3.0:1788203789.829002:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203789.829020:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187875712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203789.829083:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203789.829089:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203789.829091:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203789.829097:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5244 00000100:00100000:0.0:1788203789.829122:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187875712:192.168.201.121@tcp:8:kworker.0 00000100:00080000:3.0:1788203789.829122:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203789.829134:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203789.829142:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788203789.829162:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203789.829162:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187875840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203789.829178:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203789/real 1788203789] req@ffff8e828953b100 x1875062187875712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203789.829191:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187875712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203789.829199:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203789.829202:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203789.829205:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187875712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203789.829216:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187875840:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203789.829230:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203789.829237:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203789/real 1788203789] req@ffff8e828953a680 x1875062187875840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203789.829246:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187875840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203789.829251:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203789.829254:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203789.829256:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187875840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203790.532528:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203791.556496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203792.580510:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203793.604484:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203794.628559:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203794.948419:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203794.948428:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203794.948432:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203794.948443:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5250 00000100:00080000:3.0:1788203794.948453:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203794.948461:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203794.948495:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187875968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203794.948528:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203794.948531:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203794.948533:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203794.948538:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5250 00000100:00080000:3.0:1788203794.948542:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203794.948548:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203794.948558:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203794.948584:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187876096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203794.948596:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187875968:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788203794.948598:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203794.948607:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203794.948611:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203794.948617:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5250 00000100:00080000:3.0:1788203794.948622:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203794.948627:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203794.948633:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203794.948650:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187876224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203794.948661:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203794.948665:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203794.948666:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203794.948670:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5250 00000800:00000100:0.0:1788203794.948675:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788203794.948675:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203794.948683:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203794.948688:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203794.948699:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187876352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203794.948705:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203794.948708:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203794.948709:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203794.948712:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5250 00000100:00080000:3.0:1788203794.948714:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203794.948716:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203794.948720:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203794.948729:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203794/real 1788203794] req@ffff8e8289538a80 x1875062187875968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203794.948730:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187876480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203794.948777:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187875968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203794.948789:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203794.948794:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203794.948798:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187875968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203794.948822:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187876096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203794.948846:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203794.948861:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203794/real 1788203794] req@ffff8e8289539f80 x1875062187876096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203794.948875:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187876096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203794.948884:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203794.948888:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203794.948891:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187876096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203794.948898:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187876224:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203794.948918:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203794.948930:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203794/real 1788203794] req@ffff8e8289539500 x1875062187876224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203794.948948:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187876224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203794.948956:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203794.948959:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203794.948963:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187876224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203794.948968:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187876352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203794.948986:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203794.948995:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203794/real 1788203794] req@ffff8e8289539880 x1875062187876352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203794.949006:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187876352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203794.949039:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203794.949043:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203794.949046:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187876352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203794.949052:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187876480:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203794.949070:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203794.949080:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203794/real 1788203794] req@ffff8e828953bb80 x1875062187876480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203794.949094:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187876480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203794.949108:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203794.949111:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203794.949114:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187876480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203795.652509:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203796.676569:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203797.700478:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203798.724497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203799.748459:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203800.068417:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203800.068425:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203800.068429:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203800.068437:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5255 00000100:00080000:3.0:1788203800.068446:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203800.068452:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203800.068482:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187876608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203800.068505:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203800.068508:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203800.068510:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203800.068512:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5255 00000100:00080000:3.0:1788203800.068514:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203800.068517:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203800.068524:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203800.068534:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187876736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203800.068541:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203800.068544:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203800.068546:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203800.068551:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5255 00000100:00080000:3.0:1788203800.068555:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203800.068560:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203800.068568:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203800.068578:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187876608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203800.068586:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187876864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203800.068601:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203800.068605:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203800.068606:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203800.068610:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5255 00000100:00080000:3.0:1788203800.068612:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203800.068615:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203800.068621:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203800.068639:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187876992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203800.068649:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203800.068652:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203800.068654:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203800.068658:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5255 00000100:00080000:3.0:1788203800.068662:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203800.068666:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203800.068672:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788203800.068686:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203800.068688:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187877120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203800.068743:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203800/real 1788203800] req@ffff8e8289538380 x1875062187876608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203800.068759:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187876608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203800.068769:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203800.068772:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203800.068775:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187876608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203800.068781:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187876736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203800.068798:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203800.068806:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203800/real 1788203800] req@ffff8e828953bb80 x1875062187876736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203800.068813:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187876736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203800.068818:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203800.068821:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203800.068823:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187876736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203800.068844:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187876864:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203800.068855:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203800.068861:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203800/real 1788203800] req@ffff8e8289539880 x1875062187876864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203800.068868:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187876864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203800.068872:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203800.068874:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203800.068875:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187876864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203800.068879:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187876992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203800.068889:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203800.068894:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203800/real 1788203800] req@ffff8e8289539500 x1875062187876992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203800.068900:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187876992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203800.068905:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203800.068906:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203800.068908:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187876992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203800.068911:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187877120:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203800.068944:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203800.068952:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203800/real 1788203800] req@ffff8e8289539f80 x1875062187877120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203800.068959:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187877120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203800.068964:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203800.068966:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203800.068967:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187877120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203800.772532:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203801.796519:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203802.820495:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203803.844481:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203804.868462:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203805.188432:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203805.188445:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203805.188450:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203805.188460:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5260 00000100:00080000:1.0:1788203805.188472:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203805.188479:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203805.188520:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187877248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203805.188547:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203805.188550:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203805.188552:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203805.188557:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5260 00000100:00080000:1.0:1788203805.188562:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203805.188567:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203805.188584:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203805.188594:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187877248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203805.188599:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187877376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203805.188609:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203805.188612:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203805.188614:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203805.188618:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5260 00000100:00080000:1.0:1788203805.188623:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203805.188628:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203805.188633:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203805.188646:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187877504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203805.188655:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203805.188657:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203805.188659:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203805.188662:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5260 00000800:00000100:0.0:1788203805.188666:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788203805.188667:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203805.188672:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203805.188677:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203805.188690:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187877632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203805.188699:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203805.188703:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203805.188705:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203805.188711:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5260 00000100:00100000:0.0:1788203805.188712:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203805/real 1788203805] req@ffff8e828d722680 x1875062187877248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203805.188715:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203805.188723:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203805.188728:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203805.188739:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187877248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203805.188740:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187877760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203805.188766:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203805.188771:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203805.188774:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187877248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203805.188798:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187877376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203805.188814:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203805.188823:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203805/real 1788203805] req@ffff8e828d720000 x1875062187877376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203805.188830:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187877376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203805.188848:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203805.188851:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203805.188853:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187877376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203805.188856:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187877504:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203805.188869:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203805.188876:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203805/real 1788203805] req@ffff8e828d721500 x1875062187877504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203805.188882:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187877504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203805.188887:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203805.188889:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203805.188890:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187877504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203805.188894:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187877632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203805.188904:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203805.188908:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203805/real 1788203805] req@ffff8e828d723800 x1875062187877632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203805.188914:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187877632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203805.188919:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203805.188921:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203805.188923:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187877632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203805.188925:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187877760:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203805.188934:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203805.188945:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203805/real 1788203805] req@ffff8e828d720e00 x1875062187877760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203805.188952:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187877760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203805.188956:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203805.188958:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203805.188960:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187877760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203805.892527:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203806.916475:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203807.940518:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203808.964503:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203809.988588:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203810.308450:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203810.308461:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203810.308466:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203810.308477:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5265 00000100:00080000:3.0:1788203810.308487:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203810.308494:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203810.308530:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187877888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203810.308557:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203810.308560:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203810.308562:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203810.308566:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5265 00000100:00080000:3.0:1788203810.308570:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203810.308574:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203810.308583:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203810.308599:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187877888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203810.308599:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187878016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203810.308614:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203810.308618:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203810.308620:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203810.308623:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5265 00000100:00080000:3.0:1788203810.308627:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203810.308631:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203810.308637:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203810.308648:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187878144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203810.308662:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788203810.308668:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203810.308672:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203810.308674:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203810.308679:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5265 00000100:00080000:3.0:1788203810.308683:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203810.308688:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203810.308693:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203810.308698:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203810/real 1788203810] req@ffff8e8289538a80 x1875062187877888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203810.308707:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187878272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203810.308715:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187877888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203810.308720:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203810.308729:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203810.308733:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203810.308740:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5265 00000100:00080000:0.0:1788203810.308753:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203810.308756:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203810.308760:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:3.0:1788203810.308765:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203810.308770:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187877888:192.168.201.121@tcp:250:kworker.0 00000008:00100000:3.0:1788203810.308779:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203810.308792:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187878400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203810.308795:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187878016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203810.308823:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203810.308838:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203810/real 1788203810] req@ffff8e828953a680 x1875062187878016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203810.308849:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187878016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203810.308856:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203810.308859:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203810.308862:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187878016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203810.308867:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187878144:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203810.308884:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203810.308893:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203810/real 1788203810] req@ffff8e828953b100 x1875062187878144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203810.308904:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187878144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203810.308911:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203810.308914:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203810.308917:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187878144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203810.308922:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187878272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203810.308940:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203810.308958:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203810/real 1788203810] req@ffff8e8289539f80 x1875062187878272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203810.308966:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187878272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203810.308971:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203810.308973:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203810.308976:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187878272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203810.308990:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187878400:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203810.309018:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203810.309027:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203810/real 1788203810] req@ffff8e8289539500 x1875062187878400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203810.309034:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187878400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203810.309039:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203810.309041:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203810.309043:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187878400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203811.012507:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203812.036497:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203813.060431:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203814.084464:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203815.108482:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203815.428390:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203815.428400:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203815.428404:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203815.428414:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5270 00000100:00080000:3.0:1788203815.428425:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203815.428431:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203815.428461:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187878528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203815.428482:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203815.428484:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203815.428485:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203815.428488:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5270 00000100:00080000:3.0:1788203815.428490:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203815.428492:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203815.428498:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203815.428507:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187878656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203815.428513:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203815.428515:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203815.428517:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203815.428519:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5270 00000100:00080000:3.0:1788203815.428520:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203815.428522:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203815.428525:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203815.428530:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187878784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203815.428537:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187878528:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788203815.428537:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203815.428543:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203815.428546:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203815.428550:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5270 00000100:00080000:3.0:1788203815.428552:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203815.428554:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203815.428556:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203815.428562:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187878912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203815.428567:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203815.428568:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203815.428569:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203815.428572:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5270 00000100:00080000:3.0:1788203815.428574:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203815.428576:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203815.428578:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203815.428584:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187879040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788203815.428606:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203815.428645:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203815/real 1788203815] req@ffff8e8289539880 x1875062187878528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203815.428692:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187878528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203815.428703:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203815.428707:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203815.428710:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187878528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203815.428715:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187878656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203815.428732:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203815.428742:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203815/real 1788203815] req@ffff8e828953bb80 x1875062187878656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203815.428753:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187878656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203815.428761:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203815.428765:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203815.428768:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187878656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203815.428790:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187878784:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203815.428812:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203815.428823:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203815/real 1788203815] req@ffff8e8289538380 x1875062187878784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203815.428848:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187878784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203815.428857:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203815.428860:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203815.428864:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187878784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203815.428869:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187878912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203815.428890:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203815.428902:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203815/real 1788203815] req@ffff8e8289539500 x1875062187878912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203815.428913:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187878912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203815.428921:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203815.428924:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203815.428927:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187878912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203815.428935:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187879040:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203815.428957:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203815.428970:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203815/real 1788203815] req@ffff8e8289539f80 x1875062187879040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203815.428983:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187879040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203815.428998:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203815.429002:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203815.429006:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187879040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203816.132464:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203817.156460:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203818.180486:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203819.204482:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203820.228484:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203820.548488:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203820.548501:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203820.548508:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203820.548522:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5275 00000100:00080000:1.0:1788203820.548536:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203820.548545:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203820.548628:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187879168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203820.548661:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203820.548665:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203820.548667:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203820.548672:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5275 00000100:00080000:1.0:1788203820.548676:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203820.548681:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203820.548692:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203820.548707:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187879296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203820.548718:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187879168:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788203820.548720:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203820.548728:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203820.548732:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203820.548737:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5275 00000100:00080000:1.0:1788203820.548740:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203820.548745:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203820.548751:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203820.548766:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187879424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203820.548777:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203820.548780:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203820.548782:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203820.548786:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5275 00000100:00080000:1.0:1788203820.548801:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788203820.548806:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788203820.548810:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203820.548818:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203820.548832:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187879552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203820.548842:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203820.548846:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203820.548848:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203820.548852:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5275 00000100:00080000:1.0:1788203820.548857:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203820.548861:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203820.548867:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203820/real 1788203820] req@ffff8e828d722300 x1875062187879168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788203820.548869:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203820.548887:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187879680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203820.548894:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187879168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203820.548910:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203820.548915:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203820.548920:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187879168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203820.548951:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187879296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203820.548974:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203820.548988:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203820/real 1788203820] req@ffff8e828d722d80 x1875062187879296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203820.548999:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187879296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203820.549012:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203820.549015:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203820.549018:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187879296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203820.549023:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187879424:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203820.549041:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203820.549051:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203820/real 1788203820] req@ffff8e828d721180 x1875062187879424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203820.549061:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187879424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203820.549068:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203820.549071:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203820.549074:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187879424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203820.549078:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187879552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203820.549097:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203820.549107:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203820/real 1788203820] req@ffff8e828d720380 x1875062187879552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203820.549118:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187879552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203820.549123:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203820.549124:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203820.549127:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187879552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203820.549133:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187879680:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203820.549151:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203820.549162:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203820/real 1788203820] req@ffff8e828d720a80 x1875062187879680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203820.549172:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187879680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203820.549180:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203820.549183:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203820.549186:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187879680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203821.252462:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203822.276496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203823.300481:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203824.324472:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203825.348549:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203825.668403:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203825.668411:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203825.668415:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203825.668423:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5280 00000100:00080000:3.0:1788203825.668431:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203825.668437:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203825.668468:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187879808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203825.668508:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203825.668511:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203825.668513:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203825.668520:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5280 00000100:00080000:3.0:1788203825.668525:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203825.668532:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203825.668542:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203825.668549:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187879808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203825.668563:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187879936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203825.668575:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203825.668579:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203825.668581:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203825.668585:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5280 00000100:00080000:3.0:1788203825.668592:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203825.668597:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203825.668602:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788203825.668617:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203825.668618:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187880064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203825.668627:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203825.668629:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203825.668630:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203825.668632:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5280 00000100:00080000:3.0:1788203825.668637:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203825.668641:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203825.668645:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203825.668662:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187880192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203825.668669:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203825.668671:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203825.668673:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203825.668678:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5280 00000100:00080000:3.0:1788203825.668681:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203825.668684:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203825.668688:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203825.668702:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187880320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203825.668703:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203825/real 1788203825] req@ffff8e828953b100 x1875062187879808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203825.668724:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187879808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203825.668735:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203825.668739:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203825.668743:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187879808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203825.668767:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187879936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203825.668797:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203825.668817:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203825/real 1788203825] req@ffff8e828953a680 x1875062187879936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203825.668830:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187879936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203825.668840:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203825.668844:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203825.668847:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187879936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203825.668855:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187880064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203825.668874:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203825.668887:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203825/real 1788203825] req@ffff8e8289538a80 x1875062187880064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203825.668898:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187880064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203825.668906:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203825.668910:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203825.668913:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187880064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203825.668920:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187880192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203825.668944:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203825.668959:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203825/real 1788203825] req@ffff8e8289539f80 x1875062187880192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203825.668972:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187880192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203825.668982:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203825.668989:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203825.668997:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187880192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203825.669038:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187880320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203825.669063:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203825.669075:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203825/real 1788203825] req@ffff8e8289539500 x1875062187880320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203825.669091:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187880320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203825.669098:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203825.669101:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203825.669105:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187880320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203826.372518:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203827.396496:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203828.420451:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203829.444578:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203830.468462:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203830.788431:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203830.788442:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203830.788445:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203830.788456:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5285 00000100:00080000:2.0:1788203830.788466:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203830.788473:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203830.788517:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187880448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203830.788542:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203830.788545:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203830.788547:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203830.788551:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5285 00000100:00080000:2.0:1788203830.788555:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203830.788558:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203830.788567:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203830.788579:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187880576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203830.788590:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187880448:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203830.788591:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203830.788599:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203830.788602:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203830.788606:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5285 00000100:00080000:2.0:1788203830.788610:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203830.788613:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203830.788618:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203830.788630:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187880704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203830.788638:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203830.788642:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203830.788643:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203830.788646:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5285 00000100:00080000:2.0:1788203830.788650:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203830.788655:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:1.0:1788203830.788656:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:2.0:1788203830.788662:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203830.788675:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187880832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203830.788684:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203830.788687:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203830.788689:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203830.788693:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5285 00000100:00080000:2.0:1788203830.788696:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788203830.788698:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203830/real 1788203830] req@ffff8e82894e4a80 x1875062187880448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203830.788702:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203830.788709:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203830.788722:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187880960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203830.788724:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187880448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203830.788749:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203830.788752:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203830.788756:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187880448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203830.788777:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187880576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203830.788797:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203830.788807:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203830/real 1788203830] req@ffff8e82894e7800 x1875062187880576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203830.788824:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187880576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203830.788832:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203830.788835:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203830.788838:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187880576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203830.788841:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187880704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203830.788855:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203830.788862:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203830/real 1788203830] req@ffff8e82894e6300 x1875062187880704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203830.788868:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187880704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203830.788873:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203830.788875:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203830.788877:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187880704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203830.788880:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187880832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203830.788892:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203830.788900:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203830/real 1788203830] req@ffff8e82894e4000 x1875062187880832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203830.788910:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187880832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203830.788924:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203830.788927:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203830.788930:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187880832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203830.788935:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187880960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203830.788950:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203830.788956:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203830/real 1788203830] req@ffff8e82894e7100 x1875062187880960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203830.788962:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187880960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203830.788967:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203830.788969:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203830.788970:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187880960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203831.492480:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203832.516477:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203833.540488:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203834.564481:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203835.588473:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203835.908399:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203835.908408:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203835.908412:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203835.908420:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5290 00000100:00080000:0.0:1788203835.908428:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203835.908434:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203835.908462:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187881088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203835.908485:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203835.908488:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203835.908489:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203835.908492:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5290 00000100:00080000:0.0:1788203835.908494:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203835.908496:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203835.908502:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203835.908509:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187881216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203835.908516:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203835.908519:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203835.908520:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203835.908522:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5290 00000100:00080000:0.0:1788203835.908523:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203835.908528:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203835.908529:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187881088:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788203835.908531:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203835.908541:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187881344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203835.908548:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203835.908550:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203835.908551:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203835.908553:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5290 00000100:00080000:0.0:1788203835.908556:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203835.908558:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203835.908562:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203835.908569:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187881472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203835.908573:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203835.908575:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203835.908577:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203835.908579:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5290 00000100:00080000:0.0:1788203835.908582:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203835.908583:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203835.908586:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203835.908593:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187881600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203835.908611:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187881216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203835.908635:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187881344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203835.908650:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187881472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203835.908661:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187881600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203836.612426:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203837.636465:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203838.660447:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203838.980688:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788203838.980742:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788203838.980762:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203838.980812:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203838.980823:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203838.980834:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203838.980843:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788203838.980874:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203835/real 1788203838] req@ffff8e828793d880 x1875062187881344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203838.980912:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187881344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203838.980925:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203838.980931:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203838.980937:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187881344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203838.980945:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203835/real 1788203838] req@ffff8e828793fb80 x1875062187881472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203838.980959:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187881472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203838.980966:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203838.980983:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203838.980987:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187881472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203838.980995:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203835/real 1788203838] req@ffff8e828793d500 x1875062187881600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203838.981007:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187881600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203838.981016:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203838.981020:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203838.981023:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187881600:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203838.981030:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203835/real 1788203838] req@ffff8e828793e300 x1875062187881088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203838.981042:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187881088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203838.981049:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203838.981052:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203838.981056:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187881088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203838.981062:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203835/real 1788203838] req@ffff8e828793c380 x1875062187881216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203838.981073:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187881216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203838.981080:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203838.981084:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203838.981086:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187881216:192.168.201.121@tcp:38:kworker.0 00010000:00010000:1.0:1788203839.684520:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203840.708440:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203841.028386:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203841.028394:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203841.028398:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203841.028409:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5296 00000100:00080000:0.0:1788203841.028419:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203841.028426:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203841.028457:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187881728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203841.028489:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203841.028491:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203841.028493:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203841.028502:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5296 00000100:00080000:0.0:1788203841.028506:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203841.028511:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203841.028520:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203841.028534:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187881856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203841.028545:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187881728:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203841.028546:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203841.028554:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203841.028556:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203841.028560:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5296 00000100:00080000:0.0:1788203841.028563:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203841.028567:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203841.028572:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203841.028585:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187881984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203841.028595:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203841.028598:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203841.028600:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203841.028604:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5296 00000100:00080000:0.0:1788203841.028608:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788203841.028611:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203841.028615:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203841.028622:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203841.028634:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187882112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203841.028642:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203841.028645:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203841.028646:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203841.028649:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5296 00000100:00080000:0.0:1788203841.028653:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788203841.028658:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203841/real 1788203841] req@ffff8e828793f480 x1875062187881728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203841.028659:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203841.028666:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203841.028681:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187882240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203841.028686:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187881728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203841.028704:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203841.028709:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203841.028712:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187881728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203841.028736:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187881856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203841.028758:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203841.028770:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203841/real 1788203841] req@ffff8e828793dc00 x1875062187881856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203841.028781:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187881856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203841.028788:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203841.028791:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203841.028794:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187881856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203841.028799:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187881984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203841.028816:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203841.028827:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203841/real 1788203841] req@ffff8e828793c000 x1875062187881984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203841.028837:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187881984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203841.028843:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203841.028846:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203841.028849:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187881984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203841.028855:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187882112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203841.028872:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203841.028881:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203841/real 1788203841] req@ffff8e828793ca80 x1875062187882112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203841.028890:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187882112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203841.028898:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203841.028900:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203841.028902:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187882112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203841.028907:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187882240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203841.028922:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203841.028929:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203841/real 1788203841] req@ffff8e828793c700 x1875062187882240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203841.028938:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187882240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203841.028944:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203841.028947:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203841.028949:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187882240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203841.732510:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203842.756491:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203843.780496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203844.804477:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203845.828453:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203846.148404:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203846.148415:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203846.148419:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203846.148428:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5301 00000100:00080000:2.0:1788203846.148438:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203846.148444:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203846.148475:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187882368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203846.148500:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203846.148503:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203846.148505:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203846.148509:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5301 00000100:00080000:2.0:1788203846.148513:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203846.148517:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203846.148533:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203846.148544:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187882368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203846.148551:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187882496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203846.148562:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203846.148565:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203846.148566:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203846.148569:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5301 00000100:00080000:2.0:1788203846.148573:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203846.148577:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203846.148581:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203846.148590:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187882624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203846.148597:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203846.148600:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203846.148602:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788203846.148606:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203846.148610:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5301 00000100:00080000:2.0:1788203846.148614:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203846.148618:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203846.148624:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203846.148633:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187882752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203846.148641:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203846.148644:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203846.148646:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203846.148651:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5301 00000100:00100000:1.0:1788203846.148652:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203846/real 1788203846] req@ffff8e82894e5c00 x1875062187882368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203846.148654:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203846.148660:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203846.148668:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203846.148679:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187882368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203846.148681:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187882880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203846.148705:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203846.148711:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203846.148716:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187882368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203846.148738:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187882496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203846.148758:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203846.148768:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203846/real 1788203846] req@ffff8e82894e4380 x1875062187882496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203846.148780:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187882496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203846.148788:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203846.148791:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203846.148794:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187882496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203846.148800:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187882624:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203846.148816:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203846.148826:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203846/real 1788203846] req@ffff8e82894e5f80 x1875062187882624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203846.148845:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187882624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203846.148853:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203846.148856:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203846.148860:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187882624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203846.148865:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187882752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203846.148882:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203846.148891:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203846/real 1788203846] req@ffff8e82894e6d80 x1875062187882752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203846.148902:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187882752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203846.148910:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203846.148913:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203846.148916:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187882752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203846.148922:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187882880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203846.148937:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203846.148946:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203846/real 1788203846] req@ffff8e82894e7100 x1875062187882880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203846.148957:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187882880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203846.148970:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203846.148973:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203846.148976:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187882880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203846.852437:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203847.876435:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203848.900473:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203849.924457:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203850.948511:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203851.268461:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203851.268473:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203851.268478:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203851.268490:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5306 00000100:00080000:0.0:1788203851.268503:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203851.268511:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203851.268562:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187883008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203851.268588:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203851.268591:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203851.268592:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203851.268595:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5306 00000100:00080000:0.0:1788203851.268598:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203851.268601:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203851.268609:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203851.268619:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187883136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203851.268626:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203851.268628:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203851.268630:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203851.268632:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5306 00000100:00080000:0.0:1788203851.268634:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203851.268637:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203851.268639:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203851.268645:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187883264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203851.268651:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203851.268653:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203851.268655:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203851.268658:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5306 00000100:00080000:0.0:1788203851.268660:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203851.268666:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203851.268668:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187883008:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788203851.268671:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203851.268678:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187883392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203851.268687:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203851.268689:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203851.268690:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203851.268693:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5306 00000100:00080000:0.0:1788203851.268695:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203851.268696:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203851.268699:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203851.268707:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187883520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203851.268744:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203851.268795:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203851/real 1788203851] req@ffff8e828793f100 x1875062187883008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203851.268813:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187883008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203851.268822:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203851.268826:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203851.268828:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187883008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203851.268834:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187883136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203851.268850:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203851.268858:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203851/real 1788203851] req@ffff8e828793e680 x1875062187883136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203851.268865:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187883136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203851.268874:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203851.268876:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203851.268878:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187883136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203851.268896:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187883264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203851.268905:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203851.268911:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203851/real 1788203851] req@ffff8e828793f800 x1875062187883264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203851.268917:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187883264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203851.268921:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203851.268923:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203851.268925:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187883264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203851.268928:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187883392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203851.268937:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203851.268943:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203851/real 1788203851] req@ffff8e828793ce00 x1875062187883392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203851.268949:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187883392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203851.268953:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203851.268954:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203851.268956:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187883392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203851.268960:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187883520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203851.268968:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203851.268973:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203851/real 1788203851] req@ffff8e828793c700 x1875062187883520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203851.268978:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187883520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203851.268983:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203851.268984:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203851.268986:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187883520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203851.972524:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203852.996551:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203854.020485:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203855.044476:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203856.068454:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203856.388426:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203856.388434:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203856.388437:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203856.388446:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5311 00000100:00080000:0.0:1788203856.388456:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203856.388462:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203856.388502:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187883648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203856.388525:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203856.388527:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203856.388529:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203856.388531:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5311 00000100:00080000:0.0:1788203856.388533:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203856.388536:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203856.388541:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203856.388548:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187883776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203856.388554:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203856.388556:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203856.388558:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203856.388560:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5311 00000100:00080000:0.0:1788203856.388561:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203856.388563:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203856.388566:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203856.388572:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187883648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203856.388573:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187883904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203856.388584:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203856.388586:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203856.388587:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203856.388589:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5311 00000100:00080000:0.0:1788203856.388592:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203856.388594:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203856.388597:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203856.388604:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187884032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203856.388610:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203856.388612:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203856.388613:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203856.388614:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5311 00000100:00080000:0.0:1788203856.388623:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203856.388625:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203856.388628:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788203856.388631:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203856.388638:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187884160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203856.388668:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203856/real 1788203856] req@ffff8e828793ca80 x1875062187883648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203856.388680:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187883648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203856.388688:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203856.388690:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203856.388693:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187883648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203856.388699:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187883776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203856.388711:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203856.388718:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203856/real 1788203856] req@ffff8e828793c000 x1875062187883776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203856.388725:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187883776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203856.388730:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203856.388732:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203856.388734:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187883776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203856.388748:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187883904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203856.388757:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203856.388763:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203856/real 1788203856] req@ffff8e828793dc00 x1875062187883904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203856.388769:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187883904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203856.388776:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203856.388779:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203856.388782:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187883904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203856.388787:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187884032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203856.388805:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203856.388815:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203856/real 1788203856] req@ffff8e828793f480 x1875062187884032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203856.388826:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187884032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203856.388831:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203856.388833:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203856.388835:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187884032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203856.388838:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187884160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203856.388850:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203856.388856:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203856/real 1788203856] req@ffff8e828793c380 x1875062187884160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203856.388862:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187884160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203856.388867:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203856.388868:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203856.388870:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187884160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203857.092466:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203858.116427:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203859.140454:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203860.164470:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203861.188475:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203861.508441:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203861.508447:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203861.508450:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203861.508458:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5316 00000100:00080000:2.0:1788203861.508466:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203861.508472:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203861.508503:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187884288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203861.508536:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203861.508540:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203861.508542:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203861.508548:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5316 00000100:00080000:2.0:1788203861.508552:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203861.508557:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203861.508566:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203861.508573:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187884288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203861.508583:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187884416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203861.508592:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203861.508595:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203861.508596:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203861.508599:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5316 00000100:00080000:2.0:1788203861.508602:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203861.508604:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203861.508607:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203861.508614:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187884544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203861.508620:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203861.508623:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1788203861.508626:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203861.508626:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203861.508633:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5316 00000100:00080000:2.0:1788203861.508636:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203861.508638:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203861.508642:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203861.508649:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187884672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203861.508655:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203861.508658:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203861.508659:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203861.508661:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5316 00000100:00100000:0.0:1788203861.508664:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203861/real 1788203861] req@ffff8e82894e4000 x1875062187884288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203861.508665:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203861.508670:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203861.508676:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203861.508689:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187884288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788203861.508689:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187884800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203861.508713:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203861.508718:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203861.508724:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187884288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203861.508749:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187884416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203861.508768:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203861.508777:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203861/real 1788203861] req@ffff8e82894e6300 x1875062187884416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203861.508784:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187884416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203861.508789:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203861.508792:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203861.508795:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187884416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203861.508800:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187884544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203861.508818:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203861.508828:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203861/real 1788203861] req@ffff8e82894e7800 x1875062187884544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203861.508840:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187884544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203861.508848:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203861.508851:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203861.508867:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187884544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203861.508874:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187884672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203861.508895:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203861.508906:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203861/real 1788203861] req@ffff8e82894e4a80 x1875062187884672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203861.508918:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187884672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203861.508926:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203861.508929:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203861.508932:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187884672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203861.508939:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187884800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203861.508953:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203861.508963:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203861/real 1788203861] req@ffff8e82894e4e00 x1875062187884800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203861.508970:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187884800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203861.508975:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203861.508977:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203861.508979:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187884800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203862.212522:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203863.236465:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203864.260497:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203865.284467:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203866.308477:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788203866.628449:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203866.628460:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788203866.628464:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203866.628476:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5321 00000100:00080000:3.0:1788203866.628487:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203866.628494:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203866.628538:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187884928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203866.628568:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203866.628581:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788203866.628585:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203866.628590:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5321 00000100:00080000:3.0:1788203866.628594:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203866.628600:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203866.628611:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187884928:192.168.201.121@tcp:250:kworker.0 00000008:00100000:3.0:1788203866.628616:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203866.628638:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187885056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203866.628647:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203866.628650:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788203866.628651:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203866.628656:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5321 00000100:00080000:3.0:1788203866.628661:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203866.628666:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203866.628673:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788203866.628674:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203866.628695:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187885184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203866.628709:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203866.628712:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788203866.628713:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788203866.628717:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203866/real 1788203866] req@ffff8e8289538380 x1875062187884928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203866.628720:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5321 00000100:00080000:3.0:1788203866.628725:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203866.628732:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788203866.628740:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203866.628754:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187885312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203866.628791:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187884928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203866.628794:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788203866.628805:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203866.628810:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203866.628821:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5321 00000100:00080000:3.0:1788203866.628830:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788203866.628835:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:0.0:1788203866.628836:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788203866.628842:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788203866.628843:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203866.628851:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187884928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203866.628866:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187885440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203866.628887:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187885056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203866.628915:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203866.628924:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203866/real 1788203866] req@ffff8e828953bb80 x1875062187885056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203866.628944:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187885056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203866.628952:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203866.628955:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203866.628959:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187885056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203866.628965:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187885184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203866.628984:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203866.628992:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203866/real 1788203866] req@ffff8e8289539880 x1875062187885184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203866.628999:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187885184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203866.629007:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203866.629009:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203866.629011:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187885184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203866.629015:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187885312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203866.629037:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203866.629052:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203866/real 1788203866] req@ffff8e8289539500 x1875062187885312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203866.629063:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187885312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203866.629070:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203866.629073:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203866.629076:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187885312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203866.629090:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187885440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203866.629107:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203866.629114:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203866/real 1788203866] req@ffff8e8289539f80 x1875062187885440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203866.629121:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187885440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203866.629125:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203866.629127:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203866.629129:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187885440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203867.332474:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203868.356524:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203869.380474:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203870.404462:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203871.428484:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203871.748435:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203871.748443:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203871.748447:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203871.748454:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5326 00000100:00080000:2.0:1788203871.748464:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203871.748470:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203871.748500:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187885568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203871.748522:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203871.748528:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203871.748529:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203871.748532:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5326 00000100:00080000:2.0:1788203871.748534:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203871.748537:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203871.748544:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203871.748554:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187885696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203871.748562:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203871.748565:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203871.748567:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203871.748573:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5326 00000100:00100000:0.0:1788203871.748574:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187885568:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203871.748575:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203871.748581:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203871.748586:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203871.748593:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187885824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203871.748600:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203871.748603:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203871.748605:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203871.748608:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5326 00000100:00080000:2.0:1788203871.748611:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203871.748613:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203871.748617:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203871.748624:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187885952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203871.748630:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203871.748632:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203871.748633:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203871.748638:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5326 00000800:00000100:0.0:1788203871.748639:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203871.748644:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203871.748647:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203871.748651:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203871.748662:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187886080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788203871.748695:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203871/real 1788203871] req@ffff8e82894e6680 x1875062187885568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203871.748711:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187885568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203871.748720:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203871.748724:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203871.748729:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187885568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203871.748737:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187885696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203871.748758:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203871.748775:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203871/real 1788203871] req@ffff8e82894e5880 x1875062187885696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203871.748784:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187885696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203871.748790:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203871.748792:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203871.748794:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187885696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203871.748813:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187885824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788203871.748827:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203871.748846:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203871/real 1788203871] req@ffff8e82894e6a00 x1875062187885824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203871.748861:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187885824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203871.748870:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203871.748874:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203871.748877:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187885824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203871.748884:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187885952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203871.748901:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203871.748909:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203871/real 1788203871] req@ffff8e82894e7480 x1875062187885952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203871.748917:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187885952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203871.748922:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203871.748924:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203871.748927:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187885952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788203871.748930:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187886080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788203871.748939:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203871.748945:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203871/real 1788203871] req@ffff8e82894e4e00 x1875062187886080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203871.748957:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187886080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203871.748962:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203871.748964:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788203871.748966:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187886080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203872.452487:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203873.476470:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203874.500564:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203875.524497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203876.548493:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203876.868458:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203876.868468:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203876.868472:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203876.868481:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5331 00000100:00080000:0.0:1788203876.868491:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203876.868497:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203876.868526:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187886208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203876.868545:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203876.868547:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203876.868548:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203876.868552:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5331 00000100:00080000:0.0:1788203876.868554:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203876.868557:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203876.868564:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203876.868572:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187886208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203876.868574:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187886336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203876.868583:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203876.868585:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203876.868586:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203876.868598:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5331 00000100:00080000:0.0:1788203876.868601:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203876.868603:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203876.868606:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203876.868613:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187886464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203876.868620:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203876.868622:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203876.868623:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203876.868625:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5331 00000100:00080000:0.0:1788203876.868635:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203876.868638:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203876.868641:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788203876.868647:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203876.868649:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187886592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203876.868656:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203876.868658:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203876.868660:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203876.868665:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5331 00000100:00080000:0.0:1788203876.868670:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203876.868672:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203876.868675:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203876.868682:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187886720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203876.868695:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203876/real 1788203876] req@ffff8e828793e300 x1875062187886208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203876.868714:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187886208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203876.868721:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203876.868724:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203876.868726:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187886208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203876.868744:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187886336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203876.868761:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203876.868769:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203876/real 1788203876] req@ffff8e828793d500 x1875062187886336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203876.868776:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187886336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203876.868781:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203876.868783:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203876.868785:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187886336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203876.868790:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187886464:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203876.868799:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203876.868806:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203876/real 1788203876] req@ffff8e828793fb80 x1875062187886464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203876.868816:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187886464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203876.868824:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203876.868827:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203876.868830:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187886464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203876.868836:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187886592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203876.868855:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203876.868866:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203876/real 1788203876] req@ffff8e828793d880 x1875062187886592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203876.868877:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187886592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203876.868885:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203876.868888:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203876.868891:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187886592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203876.868897:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187886720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203876.868913:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203876.868922:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203876/real 1788203876] req@ffff8e828793c380 x1875062187886720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203876.868931:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187886720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203876.868936:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203876.868938:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203876.868940:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187886720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203877.572449:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203878.596481:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203879.620438:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203880.644535:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203881.668515:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203881.988429:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203881.988438:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203881.988442:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203881.988451:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5337 00000100:00080000:2.0:1788203881.988460:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203881.988465:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203881.988498:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187886848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203881.988547:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203881.988550:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203881.988551:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203881.988555:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5337 00000100:00080000:2.0:1788203881.988557:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203881.988561:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203881.988569:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203881.988580:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187886976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203881.988586:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203881.988589:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203881.988591:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203881.988593:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5337 00000100:00080000:2.0:1788203881.988595:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203881.988597:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203881.988602:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203881.988603:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187886848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203881.988611:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187887104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203881.988619:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203881.988621:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203881.988622:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203881.988625:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5337 00000100:00080000:2.0:1788203881.988628:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203881.988630:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203881.988633:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203881.988641:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187887232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203881.988647:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203881.988649:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203881.988650:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203881.988652:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5337 00000100:00080000:2.0:1788203881.988654:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203881.988656:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203881.988659:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203881.988667:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187887360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203881.988672:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203881.988726:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203881/real 1788203881] req@ffff8e82894e4a80 x1875062187886848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203881.988744:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187886848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203881.988754:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203881.988757:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203881.988769:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187886848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203881.988777:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187886976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203881.988800:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203881.988812:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203881/real 1788203881] req@ffff8e82894e7800 x1875062187886976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203881.988840:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187886976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203881.988848:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203881.988856:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203881.988859:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187886976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203881.988881:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187887104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203881.988900:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203881.988912:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203881/real 1788203881] req@ffff8e82894e6300 x1875062187887104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203881.988932:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187887104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203881.988940:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203881.988943:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203881.988946:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187887104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203881.988952:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187887232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203881.988969:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203881.988980:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203881/real 1788203881] req@ffff8e82894e4000 x1875062187887232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203881.988990:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187887232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203881.989004:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203881.989007:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203881.989010:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187887232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203881.989015:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187887360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203881.989033:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203881.989043:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203881/real 1788203881] req@ffff8e82894e7100 x1875062187887360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203881.989053:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187887360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203881.989070:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203881.989073:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203881.989075:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187887360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203882.692462:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203883.716547:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203884.740503:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203885.764548:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203886.788521:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203887.108500:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203887.108511:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203887.108516:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203887.108526:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5342 00000100:00080000:0.0:1788203887.108543:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203887.108551:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203887.108618:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187887488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203887.108649:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203887.108654:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203887.108656:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203887.108661:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5342 00000100:00080000:0.0:1788203887.108665:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203887.108674:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203887.108686:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203887.108703:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187887616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203887.108717:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203887.108720:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203887.108722:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203887.108727:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5342 00000100:00080000:0.0:1788203887.108733:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788203887.108734:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187887488:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203887.108743:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203887.108759:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203887.108779:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187887744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203887.108790:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203887.108792:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203887.108794:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203887.108798:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5342 00000100:00080000:0.0:1788203887.108802:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203887.108805:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203887.108810:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203887.108822:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187887872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203887.108828:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203887.108840:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203887.108843:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203887.108845:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203887.108848:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5342 00000100:00080000:0.0:1788203887.108853:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203887.108858:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203887.108865:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203887.108883:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187888000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203887.108887:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203887/real 1788203887] req@ffff8e828793f480 x1875062187887488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203887.108935:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187887488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203887.108951:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203887.108955:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203887.108959:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187887488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203887.108978:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187887616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203887.108994:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203887.109002:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203887/real 1788203887] req@ffff8e828793dc00 x1875062187887616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203887.109010:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187887616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203887.109016:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203887.109018:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203887.109020:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187887616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203887.109024:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187887744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203887.109035:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203887.109041:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203887/real 1788203887] req@ffff8e828793c000 x1875062187887744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203887.109048:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187887744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203887.109053:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203887.109055:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203887.109056:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187887744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203887.109061:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187887872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203887.109071:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203887.109076:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203887/real 1788203887] req@ffff8e828793ca80 x1875062187887872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203887.109083:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187887872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203887.109089:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203887.109090:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203887.109092:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187887872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203887.109095:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187888000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203887.109105:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203887.109110:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203887/real 1788203887] req@ffff8e828793c700 x1875062187888000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203887.109116:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187888000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203887.109129:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203887.109131:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203887.109132:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187888000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203887.812568:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203888.836507:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203889.860507:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203890.884525:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203891.908532:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203892.228393:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203892.228411:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203892.228416:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203892.228427:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5347 00000100:00080000:0.0:1788203892.228439:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203892.228446:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203892.228495:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187888128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203892.228522:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203892.228526:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203892.228527:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203892.228530:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5347 00000100:00080000:0.0:1788203892.228533:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203892.228535:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203892.228544:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203892.228558:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187888256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203892.228571:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203892.228577:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203892.228580:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788203892.228580:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187888128:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203892.228590:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5347 00000100:00080000:0.0:1788203892.228595:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203892.228600:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203892.228608:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203892.228622:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187888384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203892.228633:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203892.228637:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203892.228640:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203892.228645:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5347 00000100:00080000:0.0:1788203892.228650:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203892.228656:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:1.0:1788203892.228660:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788203892.228664:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203892.228679:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187888512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203892.228690:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203892.228694:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203892.228696:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203892.228700:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5347 00000100:00080000:0.0:1788203892.228705:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203892.228710:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203892.228717:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203892.228719:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203892/real 1788203892] req@ffff8e828793ce00 x1875062187888128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203892.228734:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187888640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203892.228740:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187888128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203892.228758:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203892.228764:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203892.228771:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187888128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203892.228801:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187888256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203892.228825:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203892.228837:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203892/real 1788203892] req@ffff8e828793f800 x1875062187888256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203892.228846:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187888256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203892.228852:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203892.228854:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203892.228857:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187888256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203892.228862:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187888384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203892.228875:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203892.228884:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203892/real 1788203892] req@ffff8e828793e680 x1875062187888384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203892.228904:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187888384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203892.228912:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203892.228916:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203892.228920:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187888384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203892.228926:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187888512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203892.228946:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203892.228955:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203892/real 1788203892] req@ffff8e828793f100 x1875062187888512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203892.228963:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187888512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203892.228968:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203892.228970:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203892.228973:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187888512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203892.228977:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187888640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203892.228988:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203892.228994:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203892/real 1788203892] req@ffff8e828793c700 x1875062187888640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203892.229002:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187888640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203892.229011:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203892.229013:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203892.229016:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187888640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203892.932516:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203893.956504:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203894.980490:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203896.004553:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203897.028455:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203897.348423:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203897.348432:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203897.348436:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203897.348444:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5352 00000100:00080000:0.0:1788203897.348454:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203897.348461:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203897.348495:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187888768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203897.348523:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203897.348526:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203897.348528:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203897.348531:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5352 00000100:00080000:0.0:1788203897.348533:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203897.348535:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203897.348543:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203897.348553:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187888896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203897.348562:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203897.348566:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788203897.348569:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187888768:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203897.348570:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203897.348575:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5352 00000100:00080000:0.0:1788203897.348581:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203897.348585:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203897.348590:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203897.348603:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187889024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203897.348612:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203897.348616:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203897.348618:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203897.348622:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5352 00000100:00080000:0.0:1788203897.348629:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203897.348632:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203897.348636:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203897.348642:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187889152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203897.348649:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203897.348653:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203897.348656:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788203897.348656:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203897.348664:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5352 00000100:00080000:0.0:1788203897.348668:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203897.348671:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203897.348676:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203897.348688:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187889280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203897.348812:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203897/real 1788203897] req@ffff8e828793ca80 x1875062187888768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203897.348845:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187888768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203897.348867:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203897.348872:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203897.348876:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187888768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203897.348902:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187888896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203897.349076:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203897.349187:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203897/real 1788203897] req@ffff8e828793c000 x1875062187888896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203897.349202:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187888896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203897.349212:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203897.349215:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203897.349219:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187888896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203897.349227:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187889024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203897.349269:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203897.349293:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203897/real 1788203897] req@ffff8e828793dc00 x1875062187889024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203897.349338:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187889024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203897.349355:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203897.349360:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203897.349365:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187889024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203897.349374:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187889152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203897.349399:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203897.349423:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203897/real 1788203897] req@ffff8e828793f480 x1875062187889152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203897.349436:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187889152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203897.349444:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203897.349447:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203897.349451:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187889152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203897.349458:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187889280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203897.349479:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203897.349491:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203897/real 1788203897] req@ffff8e828793c380 x1875062187889280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203897.349504:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187889280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203897.349512:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203897.349515:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203897.349519:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187889280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203898.052512:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203899.076541:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203900.100548:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203901.124548:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203902.148518:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203902.468475:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203902.468487:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203902.468491:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203902.468506:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5357 00000100:00080000:2.0:1788203902.468519:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203902.468527:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203902.468571:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187889408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203902.468596:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203902.468601:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203902.468603:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203902.468606:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5357 00000100:00080000:2.0:1788203902.468610:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203902.468613:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203902.468622:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203902.468632:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187889536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203902.468637:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187889408:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203902.468643:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203902.468647:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203902.468648:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203902.468651:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5357 00000100:00080000:2.0:1788203902.468653:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203902.468655:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203902.468658:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203902.468668:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187889664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203902.468674:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203902.468676:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203902.468677:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203902.468679:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5357 00000100:00080000:2.0:1788203902.468682:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203902.468685:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203902.468690:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203902.468697:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187889792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203902.468703:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203902.468706:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203902.468707:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203902.468710:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5357 00000100:00080000:2.0:1788203902.468712:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203902.468714:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203902.468717:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203902.468726:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187889920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203902.468770:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187889536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203902.468796:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187889664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203902.468812:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187889792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203902.468827:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187889920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203903.172537:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203904.196505:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203905.220543:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203905.540962:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788203905.541054:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788203905.541078:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203905.541158:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203905.541170:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203905.541179:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203905.541188:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788203905.541303:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203902/real 1788203905] req@ffff8e82894e5f80 x1875062187889536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203905.541391:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187889536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203905.541407:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203905.541414:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203905.541420:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187889536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203905.541430:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203902/real 1788203905] req@ffff8e82894e4380 x1875062187889664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203905.541445:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187889664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203905.541454:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203905.541459:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203905.541463:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187889664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203905.541471:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203902/real 1788203905] req@ffff8e82894e5c00 x1875062187889792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203905.541486:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187889792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203905.541495:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203905.541499:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203905.541503:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187889792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203905.541512:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203902/real 1788203905] req@ffff8e82894e7100 x1875062187889920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203905.541525:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187889920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203905.541534:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203905.541538:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203905.541543:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187889920:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203905.541550:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203902/real 1788203905] req@ffff8e82894e6d80 x1875062187889408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203905.541565:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187889408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203905.541574:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203905.541578:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203905.541582:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187889408:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788203906.244503:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203907.268512:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203907.588458:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203907.588469:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203907.588475:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203907.588487:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5362 00000100:00080000:2.0:1788203907.588500:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203907.588510:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203907.588560:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187890048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203907.588599:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203907.588604:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203907.588606:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203907.588611:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5362 00000100:00080000:2.0:1788203907.588617:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203907.588622:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203907.588636:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203907.588654:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187890176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203907.588669:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203907.588673:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203907.588677:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788203907.588679:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187890048:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203907.588684:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5362 00000100:00080000:2.0:1788203907.588693:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203907.588698:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203907.588705:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203907.588722:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187890304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203907.588733:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203907.588737:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203907.588739:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203907.588742:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5362 00000100:00080000:2.0:1788203907.588747:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203907.588751:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203907.588757:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203907.588770:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187890432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203907.588786:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203907.588797:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203907.588801:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203907.588803:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203907.588807:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5362 00000100:00080000:2.0:1788203907.588812:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203907.588817:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203907.588823:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203907.588837:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187890560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203907.588879:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203907/real 1788203907] req@ffff8e82894e4000 x1875062187890048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203907.588906:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187890048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203907.588920:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203907.588926:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203907.588931:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187890048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203907.588971:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187890176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203907.589000:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203907.589016:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203907/real 1788203907] req@ffff8e82894e6300 x1875062187890176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203907.589029:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187890176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203907.589039:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203907.589043:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203907.589047:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187890176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203907.589095:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187890304:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203907.589119:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203907.589133:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203907/real 1788203907] req@ffff8e82894e7800 x1875062187890304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203907.589145:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187890304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203907.589152:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203907.589155:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203907.589159:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187890304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203907.589165:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187890432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203907.589185:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203907.589196:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203907/real 1788203907] req@ffff8e82894e4a80 x1875062187890432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203907.589207:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187890432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203907.589215:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203907.589219:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203907.589223:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187890432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203907.589229:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187890560:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203907.589249:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203907.589261:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203907/real 1788203907] req@ffff8e82894e4e00 x1875062187890560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203907.589275:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187890560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203907.589289:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203907.589293:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203907.589297:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187890560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203908.292595:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203909.316446:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203910.340490:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203911.364448:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203912.388501:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203912.708404:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203912.708413:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203912.708416:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203912.708424:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5367 00000100:00080000:2.0:1788203912.708433:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203912.708439:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203912.708469:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187890688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203912.708491:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203912.708494:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203912.708495:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203912.708498:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5367 00000100:00080000:2.0:1788203912.708501:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203912.708503:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203912.708511:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203912.708520:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187890816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203912.708528:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203912.708530:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203912.708531:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203912.708534:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5367 00000100:00080000:2.0:1788203912.708536:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203912.708540:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203912.708545:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187890688:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788203912.708545:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203912.708554:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187890944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203912.708561:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203912.708564:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203912.708565:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203912.708567:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5367 00000100:00080000:2.0:1788203912.708569:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203912.708572:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203912.708575:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203912.708602:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187891072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203912.708609:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203912.708611:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203912.708613:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203912.708615:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5367 00000100:00080000:2.0:1788203912.708623:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203912.708626:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203912.708630:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788203912.708635:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203912.708643:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187891200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203912.708706:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203912/real 1788203912] req@ffff8e82894e7480 x1875062187890688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203912.708721:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187890688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203912.708730:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203912.708733:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203912.708737:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187890688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203912.708750:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187890816:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203912.708784:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203912.708793:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203912/real 1788203912] req@ffff8e82894e6a00 x1875062187890816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203912.708806:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187890816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203912.708812:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203912.708814:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203912.708816:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187890816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203912.708853:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187890944:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203912.708872:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203912.708884:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203912/real 1788203912] req@ffff8e82894e5880 x1875062187890944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203912.708897:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187890944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203912.708921:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203912.708925:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203912.708927:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187890944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203912.708932:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187891072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203912.708956:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203912.708964:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203912/real 1788203912] req@ffff8e82894e6680 x1875062187891072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203912.709007:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187891072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203912.709017:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203912.709019:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203912.709021:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187891072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203912.709025:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187891200:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203912.709039:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203912.709046:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203912/real 1788203912] req@ffff8e82894e4e00 x1875062187891200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203912.709077:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187891200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203912.709083:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203912.709085:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203912.709087:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187891200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203913.412453:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203914.436488:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203915.460462:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203916.484453:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203917.508479:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203917.828422:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203917.828432:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203917.828436:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203917.828445:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5372 00000100:00080000:2.0:1788203917.828454:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203917.828460:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203917.828497:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187891328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203917.828528:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203917.828531:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203917.828533:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203917.828536:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5372 00000100:00080000:2.0:1788203917.828540:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203917.828544:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203917.828552:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203917.828564:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187891456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203917.828576:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203917.828579:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203917.828580:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203917.828583:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5372 00000100:00080000:2.0:1788203917.828585:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203917.828587:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203917.828592:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203917.828593:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187891328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203917.828602:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187891584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203917.828614:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203917.828617:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203917.828620:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203917.828624:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5372 00000100:00080000:2.0:1788203917.828628:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203917.828632:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203917.828638:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203917.828651:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187891712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203917.828661:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203917.828663:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203917.828664:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203917.828667:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5372 00000100:00080000:2.0:1788203917.828671:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788203917.828677:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203917.828678:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203917.828685:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203917.828696:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187891840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203917.828735:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203917/real 1788203917] req@ffff8e82894e4a80 x1875062187891328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203917.828757:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187891328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203917.828776:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203917.828780:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203917.828784:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187891328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203917.828793:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187891456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203917.828819:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203917.828833:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203917/real 1788203917] req@ffff8e82894e7800 x1875062187891456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203917.828847:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187891456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203917.828856:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203917.828859:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203917.828863:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187891456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203917.828907:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187891584:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203917.828931:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203917.828945:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203917/real 1788203917] req@ffff8e82894e6300 x1875062187891584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203917.828958:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187891584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203917.828967:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203917.828971:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203917.828975:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187891584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203917.828982:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187891712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203917.829006:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203917.829019:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203917/real 1788203917] req@ffff8e82894e4000 x1875062187891712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203917.829033:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187891712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203917.829042:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203917.829045:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203917.829049:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187891712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203917.829055:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187891840:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203917.829076:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203917.829089:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203917/real 1788203917] req@ffff8e82894e6d80 x1875062187891840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203917.829102:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187891840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203917.829110:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203917.829114:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203917.829117:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187891840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203918.532501:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203919.556488:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203920.580545:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203921.604461:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203922.628476:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203922.948476:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203922.948489:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203922.948494:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203922.948505:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5378 00000100:00080000:0.0:1788203922.948516:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203922.948524:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203922.948563:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187891968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203922.948596:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203922.948601:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203922.948603:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203922.948608:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5378 00000100:00080000:0.0:1788203922.948612:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203922.948617:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203922.948627:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203922.948643:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187892096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203922.948662:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187891968:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203922.948670:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203922.948680:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203922.948683:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203922.948688:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5378 00000100:00080000:0.0:1788203922.948693:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203922.948698:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203922.948704:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203922.948720:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187892224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203922.948732:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788203922.948733:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203922.948740:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203922.948742:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203922.948748:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5378 00000100:00080000:0.0:1788203922.948753:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203922.948758:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203922.948764:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203922.948778:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187892352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203922.948784:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203922/real 1788203922] req@ffff8e828793d880 x1875062187891968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203922.948814:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203922.948818:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203922.948820:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203922.948829:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5378 00000100:00080000:0.0:1788203922.948837:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203922.948844:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203922.948849:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187891968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788203922.948851:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788203922.948866:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788203922.948872:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187892480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203922.948873:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203922.948878:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187891968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203922.948905:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187892096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203922.948931:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203922.948946:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203922/real 1788203922] req@ffff8e828793fb80 x1875062187892096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203922.948959:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187892096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203922.948968:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203922.948971:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203922.948974:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187892096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203922.948978:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187892224:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203922.948994:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203922.949001:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203922/real 1788203922] req@ffff8e828793d500 x1875062187892224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203922.949014:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187892224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203922.949019:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203922.949021:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203922.949023:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187892224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203922.949026:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187892352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203922.949037:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203922.949043:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203922/real 1788203922] req@ffff8e828793e300 x1875062187892352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203922.949050:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187892352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203922.949054:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203922.949056:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203922.949058:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187892352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203922.949061:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187892480:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203922.949070:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203922.949075:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203922/real 1788203922] req@ffff8e828793c380 x1875062187892480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203922.949082:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187892480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203922.949091:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203922.949093:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203922.949095:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187892480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788203923.652471:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203924.676482:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203925.700452:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203926.724523:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203927.748488:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203928.068432:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203928.068443:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203928.068448:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203928.068458:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5383 00000100:00080000:2.0:1788203928.068470:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203928.068477:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203928.068514:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187892608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203928.068541:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203928.068545:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203928.068546:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203928.068550:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5383 00000100:00080000:2.0:1788203928.068554:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203928.068559:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203928.068569:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203928.068579:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187892608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203928.068586:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187892736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203928.068600:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203928.068607:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203928.068609:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203928.068613:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5383 00000100:00080000:2.0:1788203928.068617:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203928.068620:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203928.068625:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203928.068636:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187892864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203928.068648:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203928.068650:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203928.068655:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203928.068657:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203928.068663:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5383 00000100:00080000:2.0:1788203928.068668:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203928.068673:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203928.068677:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203928.068689:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187892992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203928.068695:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203928/real 1788203928] req@ffff8e82894e7100 x1875062187892608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203928.068714:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203928.068723:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203928.068726:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788203928.068730:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187892608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203928.068734:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5383 00000100:00080000:2.0:1788203928.068743:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203928.068749:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788203928.068750:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:1.0:1788203928.068759:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000008:00100000:2.0:1788203928.068761:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203928.068765:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187892608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203928.068778:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187893120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203928.068795:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187892736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203928.068821:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203928.068834:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203928/real 1788203928] req@ffff8e82894e5c00 x1875062187892736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203928.068846:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187892736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203928.068861:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203928.068865:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203928.068868:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187892736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203928.068874:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187892864:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203928.068892:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203928.068902:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203928/real 1788203928] req@ffff8e82894e4380 x1875062187892864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203928.068914:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187892864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203928.068922:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203928.068925:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203928.068929:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187892864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203928.068944:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187892992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203928.068963:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203928.068973:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203928/real 1788203928] req@ffff8e82894e5f80 x1875062187892992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203928.068984:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187892992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203928.068992:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203928.068995:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203928.068999:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187892992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203928.069005:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187893120:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203928.069022:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203928.069032:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203928/real 1788203928] req@ffff8e82894e6d80 x1875062187893120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203928.069044:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187893120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203928.069052:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203928.069055:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203928.069058:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187893120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203928.772492:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203929.796487:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203930.820539:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203931.844439:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203932.868554:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203933.188420:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203933.188428:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203933.188432:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203933.188449:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5388 00000100:00080000:2.0:1788203933.188458:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203933.188465:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203933.188496:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187893248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203933.188524:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203933.188529:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203933.188531:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203933.188535:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5388 00000100:00080000:2.0:1788203933.188539:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203933.188544:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203933.188554:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203933.188570:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187893376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203933.188577:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187893248:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203933.188589:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203933.188593:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203933.188595:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203933.188601:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5388 00000100:00080000:2.0:1788203933.188605:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203933.188608:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203933.188634:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203933.188645:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187893504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203933.188653:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788203933.188658:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203933.188658:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203933.188662:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203933.188666:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5388 00000100:00080000:2.0:1788203933.188671:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203933.188674:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203933.188678:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203933.188689:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187893632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203933.188699:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203933.188702:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203933.188705:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203933.188709:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5388 00000100:00100000:1.0:1788203933.188712:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203933/real 1788203933] req@ffff8e82894e4000 x1875062187893248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788203933.188722:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203933.188730:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203933.188732:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187893248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203933.188742:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203933.188747:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000008:00100000:2.0:1788203933.188748:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203933.188751:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187893248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203933.188768:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187893760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203933.188777:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187893376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203933.188809:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203933.188826:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203933/real 1788203933] req@ffff8e82894e6300 x1875062187893376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203933.188843:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187893376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203933.188852:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203933.188856:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203933.188861:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187893376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203933.188869:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187893504:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203933.188894:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203933.188908:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203933/real 1788203933] req@ffff8e82894e7800 x1875062187893504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203933.188922:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187893504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203933.188931:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203933.188934:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203933.188937:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187893504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203933.188945:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187893632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203933.188975:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203933.189002:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203933/real 1788203933] req@ffff8e82894e4a80 x1875062187893632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203933.189018:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187893632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203933.189026:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203933.189029:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203933.189033:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187893632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203933.189051:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187893760:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203933.189076:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203933.189089:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203933/real 1788203933] req@ffff8e82894e4e00 x1875062187893760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203933.189102:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187893760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203933.189108:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203933.189113:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203933.189115:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187893760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203933.892455:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203934.916510:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788203935.940491:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203936.964573:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203937.988493:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203938.308385:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203938.308393:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203938.308396:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203938.308405:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5393 00000100:00080000:2.0:1788203938.308414:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203938.308420:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203938.308451:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187893888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203938.308474:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203938.308477:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203938.308478:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203938.308481:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5393 00000100:00080000:2.0:1788203938.308483:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203938.308486:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203938.308493:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203938.308500:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187894016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203938.308507:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203938.308509:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203938.308510:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203938.308513:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5393 00000100:00080000:2.0:1788203938.308515:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203938.308517:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203938.308532:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203938.308544:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187894144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203938.308553:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788203938.308554:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187893888:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203938.308558:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203938.308562:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203938.308567:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5393 00000100:00080000:2.0:1788203938.308569:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203938.308572:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203938.308577:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203938.308585:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187894272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203938.308591:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203938.308594:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203938.308596:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203938.308598:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5393 00000100:00080000:2.0:1788203938.308601:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203938.308604:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203938.308606:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203938.308613:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187894400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203938.308628:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203938.308670:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203938/real 1788203938] req@ffff8e82894e6680 x1875062187893888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203938.308697:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187893888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203938.308709:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203938.308714:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203938.308718:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187893888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203938.308726:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187894016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203938.308752:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203938.308766:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203938/real 1788203938] req@ffff8e82894e5880 x1875062187894016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203938.308779:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187894016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203938.308789:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203938.308793:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203938.308797:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187894016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203938.308837:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187894144:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203938.308860:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203938.308873:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203938/real 1788203938] req@ffff8e82894e6a00 x1875062187894144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203938.308886:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187894144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203938.308901:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203938.308905:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203938.308908:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187894144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203938.308915:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187894272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203938.308936:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203938.308948:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203938/real 1788203938] req@ffff8e82894e7480 x1875062187894272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203938.308960:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187894272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203938.308969:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203938.308973:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203938.308976:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187894272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203938.308983:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187894400:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203938.309002:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203938.309024:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203938/real 1788203938] req@ffff8e82894e4e00 x1875062187894400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203938.309036:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187894400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203938.309044:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203938.309048:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203938.309052:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187894400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203939.012558:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203940.036528:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203941.060538:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203942.084402:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203943.108491:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203943.428463:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203943.428475:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203943.428482:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203943.428496:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5398 00000100:00080000:2.0:1788203943.428509:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203943.428519:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203943.428580:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187894528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203943.428611:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203943.428614:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203943.428616:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203943.428619:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5398 00000100:00080000:2.0:1788203943.428622:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203943.428625:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203943.428635:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203943.428645:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187894656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203943.428653:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203943.428656:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203943.428657:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203943.428660:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5398 00000100:00080000:2.0:1788203943.428661:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203943.428664:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203943.428668:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203943.428670:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187894528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203943.428677:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187894784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203943.428695:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203943.428699:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203943.428700:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203943.428706:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5398 00000100:00080000:2.0:1788203943.428711:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203943.428715:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203943.428721:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203943.428735:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187894912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788203943.428744:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788203943.428748:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203943.428754:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203943.428756:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203943.428762:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5398 00000100:00080000:2.0:1788203943.428767:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203943.428772:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203943.428778:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203943.428795:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187895040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788203943.428801:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203943/real 1788203943] req@ffff8e82894e4a80 x1875062187894528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203943.428824:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187894528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203943.428833:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203943.428836:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203943.428840:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187894528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203943.428848:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187894656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203943.428864:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203943.428873:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203943/real 1788203943] req@ffff8e82894e7800 x1875062187894656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203943.428881:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187894656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203943.428887:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203943.428888:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203943.428892:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187894656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203943.428917:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187894784:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788203943.428937:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203943.428950:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203943/real 1788203943] req@ffff8e82894e6300 x1875062187894784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203943.428963:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187894784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203943.428972:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203943.428975:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203943.428979:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187894784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203943.428985:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187894912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203943.429007:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203943.429020:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203943/real 1788203943] req@ffff8e82894e4000 x1875062187894912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203943.429030:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187894912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203943.429035:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203943.429038:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203943.429040:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187894912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203943.429044:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187895040:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788203943.429058:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788203943.429066:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203943/real 1788203943] req@ffff8e82894e6d80 x1875062187895040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203943.429073:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187895040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203943.429078:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203943.429080:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203943.429082:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187895040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203944.132513:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203945.156517:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203946.180528:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203947.204494:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203948.228558:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203948.548470:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203948.548483:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203948.548489:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203948.548503:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5403 00000100:00080000:1.0:1788203948.548517:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203948.548526:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203948.548582:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187895168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203948.548619:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203948.548624:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203948.548626:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203948.548632:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5403 00000100:00080000:1.0:1788203948.548637:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203948.548644:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203948.548657:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203948.548663:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187895168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203948.548686:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187895296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203948.548699:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203948.548704:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203948.548706:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203948.548710:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5403 00000100:00080000:1.0:1788203948.548715:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203948.548721:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203948.548727:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203948.548740:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187895424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203948.548752:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788203948.548757:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788203948.548759:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203948.548763:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203948.548770:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5403 00000100:00080000:1.0:1788203948.548775:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203948.548780:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203948.548787:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203948.548802:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187895552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203948.548826:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203948/real 1788203948] req@ffff8e828d720a80 x1875062187895168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203948.548828:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203948.548838:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203948.548844:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203948.548851:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5403 00000100:00080000:1.0:1788203948.548856:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788203948.548862:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187895168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203948.548863:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203948.548875:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788203948.548880:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203948.548888:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203948.548893:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187895680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203948.548898:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187895168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203948.548938:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187895296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203948.548966:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203948.548981:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203948/real 1788203948] req@ffff8e828d720380 x1875062187895296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203948.548994:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187895296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203948.549003:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203948.549007:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203948.549010:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187895296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203948.549017:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187895424:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203948.549038:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203948.549050:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203948/real 1788203948] req@ffff8e828d721180 x1875062187895424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203948.549063:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187895424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203948.549071:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203948.549075:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203948.549078:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187895424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203948.549085:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187895552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203948.549108:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203948.549121:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203948/real 1788203948] req@ffff8e828d722d80 x1875062187895552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203948.549133:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187895552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203948.549142:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203948.549145:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203948.549149:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187895552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203948.549156:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187895680:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203948.549177:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203948.549188:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203948/real 1788203948] req@ffff8e828d722300 x1875062187895680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203948.549200:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187895680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203948.549209:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203948.549212:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203948.549215:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187895680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203949.252506:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203950.276524:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203951.300516:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203952.324465:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203953.348471:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203953.668396:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203953.668408:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203953.668412:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203953.668423:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5408 00000100:00080000:1.0:1788203953.668433:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203953.668439:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203953.668474:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187895808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203953.668499:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203953.668501:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203953.668503:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203953.668506:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5408 00000100:00080000:1.0:1788203953.668509:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203953.668512:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203953.668519:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203953.668529:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187895936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203953.668536:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203953.668539:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203953.668540:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203953.668542:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5408 00000100:00080000:1.0:1788203953.668544:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203953.668549:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203953.668551:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187895808:192.168.201.121@tcp:250:kworker.0 00000008:00100000:1.0:1788203953.668552:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203953.668563:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187896064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203953.668570:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203953.668572:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203953.668574:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203953.668576:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5408 00000100:00080000:1.0:1788203953.668578:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203953.668580:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203953.668583:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203953.668590:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187896192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203953.668600:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203953.668603:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203953.668604:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203953.668607:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5408 00000100:00080000:1.0:1788203953.668609:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203953.668611:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:3.0:1788203953.668616:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:1.0:1788203953.668617:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203953.668627:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187896320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203953.668664:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203953/real 1788203953] req@ffff8e828d720e00 x1875062187895808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203953.668679:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187895808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203953.668689:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203953.668692:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203953.668696:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187895808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203953.668701:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187895936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203953.668716:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203953.668725:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203953/real 1788203953] req@ffff8e828d723800 x1875062187895936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203953.668733:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187895936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203953.668738:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203953.668740:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203953.668742:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187895936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203953.668762:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187896064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203953.668772:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203953.668779:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203953/real 1788203953] req@ffff8e828d721500 x1875062187896064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203953.668786:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187896064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203953.668791:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203953.668793:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203953.668794:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187896064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203953.668799:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187896192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203953.668809:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203953.668815:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203953/real 1788203953] req@ffff8e828d720000 x1875062187896192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203953.668822:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187896192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203953.668827:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203953.668829:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203953.668830:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187896192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203953.668834:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187896320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203953.668855:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203953.668861:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203953/real 1788203953] req@ffff8e828d722680 x1875062187896320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203953.668868:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187896320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203953.668873:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203953.668875:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203953.668877:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187896320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203954.372498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203955.396498:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203956.420486:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203957.444441:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203958.468460:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203958.788441:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203958.788449:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203958.788454:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203958.788466:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5413 00000100:00080000:0.0:1788203958.788477:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203958.788485:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203958.788527:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187896448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203958.788555:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203958.788559:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203958.788561:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203958.788566:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5413 00000100:00080000:0.0:1788203958.788570:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203958.788575:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203958.788586:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203958.788597:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187896448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203958.788605:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187896576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203958.788614:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203958.788617:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203958.788618:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203958.788621:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5413 00000100:00080000:0.0:1788203958.788625:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203958.788627:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203958.788631:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203958.788640:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187896704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203958.788647:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203958.788650:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203958.788651:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788203958.788655:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203958.788656:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5413 00000100:00080000:0.0:1788203958.788661:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203958.788664:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203958.788668:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203958.788676:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187896832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203958.788681:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203958.788684:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203958.788685:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203958.788689:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5413 00000100:00100000:3.0:1788203958.788694:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203958/real 1788203958] req@ffff8e828793f480 x1875062187896448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203958.788697:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203958.788705:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203958.788711:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203958.788720:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187896448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203958.788721:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187896960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203958.788749:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203958.788756:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203958.788761:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187896448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203958.788785:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187896576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203958.788811:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203958.788826:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203958/real 1788203958] req@ffff8e828793dc00 x1875062187896576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203958.788837:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187896576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203958.788843:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203958.788846:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203958.788848:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187896576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203958.788852:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187896704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203958.788868:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203958.788875:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203958/real 1788203958] req@ffff8e828793c000 x1875062187896704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203958.788883:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187896704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203958.788895:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203958.788898:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203958.788901:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187896704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203958.788907:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187896832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203958.788927:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203958.788938:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203958/real 1788203958] req@ffff8e828793ca80 x1875062187896832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203958.788951:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187896832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203958.788956:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203958.788958:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203958.788961:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187896832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203958.788965:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187896960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203958.788980:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203958.788986:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203958/real 1788203958] req@ffff8e828793c700 x1875062187896960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203958.788994:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187896960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203958.788998:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203958.789000:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203958.789003:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187896960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203959.492513:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203960.516449:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203961.540493:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203962.564491:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203963.588478:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203963.908434:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203963.908443:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203963.908447:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203963.908455:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5418 00000100:00080000:2.0:1788203963.908464:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203963.908470:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203963.908503:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187897088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203963.908525:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203963.908529:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203963.908531:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203963.908535:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5418 00000100:00080000:2.0:1788203963.908539:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203963.908543:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203963.908552:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203963.908565:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187897216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203963.908572:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187897088:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788203963.908576:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203963.908579:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203963.908581:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203963.908584:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5418 00000100:00080000:2.0:1788203963.908587:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203963.908591:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203963.908596:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203963.908604:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187897344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203963.908612:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203963.908615:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203963.908617:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203963.908620:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5418 00000100:00080000:2.0:1788203963.908623:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203963.908626:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203963.908631:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788203963.908631:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788203963.908639:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187897472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203963.908647:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203963.908650:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203963.908652:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203963.908655:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5418 00000100:00080000:2.0:1788203963.908658:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203963.908661:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203963.908671:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203963/real 1788203963] req@ffff8e82894e5f80 x1875062187897088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788203963.908675:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203963.908686:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187897600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203963.908688:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187897088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203963.908699:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203963.908703:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203963.908707:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187897088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203963.908729:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187897216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203963.908749:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203963.908759:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203963/real 1788203963] req@ffff8e82894e4380 x1875062187897216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203963.908767:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187897216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203963.908772:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203963.908775:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203963.908777:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187897216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203963.908781:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187897344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203963.908793:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203963.908804:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203963/real 1788203963] req@ffff8e82894e5c00 x1875062187897344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203963.908816:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187897344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203963.908825:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203963.908828:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203963.908832:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187897344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203963.908839:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187897472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203963.908857:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203963.908865:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203963/real 1788203963] req@ffff8e82894e7100 x1875062187897472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203963.908873:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187897472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203963.908878:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203963.908880:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203963.908882:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187897472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203963.908885:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187897600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203963.908895:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203963.908901:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203963/real 1788203963] req@ffff8e82894e6d80 x1875062187897600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203963.908908:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187897600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203963.908914:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203963.908916:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203963.908918:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187897600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203964.612457:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203965.636535:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203966.660523:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203967.684474:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203968.708472:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203969.028452:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203969.028463:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203969.028468:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203969.028479:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5424 00000100:00080000:2.0:1788203969.028490:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203969.028498:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203969.028538:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187897728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203969.028569:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203969.028573:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203969.028575:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203969.028580:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5424 00000100:00080000:2.0:1788203969.028584:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203969.028596:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203969.028607:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203969.028621:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187897728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788203969.028624:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187897856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203969.028641:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203969.028645:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203969.028648:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203969.028652:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5424 00000100:00080000:2.0:1788203969.028657:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203969.028662:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203969.028668:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203969.028682:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187897984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203969.028693:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203969.028697:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203969.028699:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203969.028704:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5424 00000100:00080000:2.0:1788203969.028710:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203969.028715:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203969.028722:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203969.028723:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187897856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788203969.028739:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187898112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203969.028752:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203969.028756:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203969.028759:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203969.028766:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5424 00000100:00100000:3.0:1788203969.028766:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187897984:192.168.201.121@tcp:38:kworker.0 00000100:00080000:2.0:1788203969.028773:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203969.028779:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203969.028785:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203969.028799:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187898240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203969.028804:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187898112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203969.028831:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187898240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203969.732492:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203970.756570:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203971.780599:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788203972.100810:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788203972.100885:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788203972.100909:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203972.100984:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203972.100998:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203972.101008:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788203972.101017:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788203972.101119:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203969/real 1788203972] req@ffff8e82894e4e00 x1875062187898240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203972.101152:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187898240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203972.101169:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203972.101175:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203972.101179:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187898240:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203972.101191:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203969/real 1788203972] req@ffff8e82894e4a80 x1875062187898112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203972.101206:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187898112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203972.101214:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203972.101217:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203972.101221:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187898112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788203972.101229:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203969/real 1788203972] req@ffff8e82894e6300 x1875062187897856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203972.101244:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187897856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203972.101254:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203972.101259:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203972.101262:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187897856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203972.101288:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203969/real 1788203972] req@ffff8e82894e7800 x1875062187897984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203972.101303:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187897984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203972.101355:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203972.101360:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203972.101364:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187897984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788203972.101372:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203969/real 1788203972] req@ffff8e82894e4000 x1875062187897728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788203972.101389:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187897728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203972.101399:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788203972.101403:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788203972.101406:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187897728:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788203972.804457:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203973.828494:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203974.148464:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203974.148473:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203974.148477:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203974.148485:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5429 00000100:00080000:1.0:1788203974.148494:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203974.148507:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203974.148537:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187898368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203974.148567:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203974.148572:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203974.148574:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203974.148578:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5429 00000100:00080000:1.0:1788203974.148582:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203974.148588:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203974.148603:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203974.148617:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187898368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203974.148621:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187898496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203974.148635:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203974.148639:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788203974.148642:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203974.148646:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5429 00000100:00080000:1.0:1788203974.148650:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203974.148655:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203974.148660:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203974.148673:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187898624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203974.148684:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788203974.148690:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788203974.148691:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203974.148695:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203974.148701:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5429 00000100:00080000:1.0:1788203974.148707:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203974.148713:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203974.148718:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203974.148736:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187898752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203974.148739:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203974/real 1788203974] req@ffff8e828d722680 x1875062187898368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203974.148756:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203974.148763:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203974.148765:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203974.148771:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5429 00000100:00100000:3.0:1788203974.148774:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187898368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788203999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788203974.148776:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203974.148785:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:3.0:1788203974.148791:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1788203974.148797:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788203974.148798:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203974.148806:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187898368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788203974.148816:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187898880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203974.148838:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187898496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203974.148865:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203974.148879:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203974/real 1788203974] req@ffff8e828d720000 x1875062187898496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203974.148892:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187898496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203974.148914:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203974.148918:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203974.148922:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187898496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203974.148928:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187898624:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203974.148951:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203974.148964:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203974/real 1788203974] req@ffff8e828d721500 x1875062187898624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203974.148976:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187898624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788203999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203974.148985:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203974.148989:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203974.148994:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187898624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203974.149000:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187898752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203974.149017:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203974.149028:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203974/real 1788203974] req@ffff8e828d723800 x1875062187898752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203974.149038:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187898752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203974.149045:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203974.149049:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203974.149051:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187898752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203974.149057:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187898880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203974.149073:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203974.149083:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203974/real 1788203974] req@ffff8e828d720e00 x1875062187898880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203974.149093:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187898880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788203999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203974.149099:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203974.149102:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203974.149105:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187898880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203974.852429:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203975.876444:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203976.900511:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203977.924499:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203978.948477:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788203979.268429:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203979.268445:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788203979.268451:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203979.268464:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5434 00000100:00080000:2.0:1788203979.268475:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203979.268483:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788203979.268527:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187899008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203979.268553:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203979.268559:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788203979.268561:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203979.268564:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5434 00000100:00080000:2.0:1788203979.268567:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203979.268570:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203979.268580:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203979.268590:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187899136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203979.268598:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203979.268601:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788203979.268602:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203979.268605:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5434 00000100:00080000:2.0:1788203979.268607:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203979.268612:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203979.268615:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187899008:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788203979.268616:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788203979.268626:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187899264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203979.268634:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203979.268636:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788203979.268638:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203979.268647:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5434 00000100:00080000:2.0:1788203979.268649:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203979.268652:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203979.268657:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203979.268666:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187899392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788203979.268674:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788203979.268677:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788203979.268678:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788203979.268681:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5434 00000100:00080000:2.0:1788203979.268686:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788203979.268689:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788203979.268692:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788203979.268703:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187899520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203979.268709:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203979.268761:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203979/real 1788203979] req@ffff8e82894e7480 x1875062187899008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203979.268779:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187899008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203979.268789:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203979.268792:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203979.268796:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187899008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203979.268801:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187899136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203979.268838:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203979.268849:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203979/real 1788203979] req@ffff8e82894e6a00 x1875062187899136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203979.268858:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187899136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203979.268865:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203979.268867:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203979.268869:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187899136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203979.268893:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187899264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203979.268906:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203979.268913:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203979/real 1788203979] req@ffff8e82894e5880 x1875062187899264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203979.268921:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187899264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203979.268931:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203979.268933:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203979.268935:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187899264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203979.268939:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187899392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203979.268950:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203979.268955:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203979/real 1788203979] req@ffff8e82894e6680 x1875062187899392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203979.268963:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187899392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203979.268968:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203979.268970:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203979.268972:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187899392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203979.268975:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187899520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203979.268985:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203979.268990:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203979/real 1788203979] req@ffff8e82894e4000 x1875062187899520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203979.268998:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187899520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203979.269003:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203979.269004:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203979.269007:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187899520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203979.972501:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203980.996532:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203982.020483:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203983.044513:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203984.068495:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203984.388425:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203984.388434:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203984.388438:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203984.388447:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5439 00000100:00080000:0.0:1788203984.388456:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203984.388461:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203984.388494:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187899648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203984.388517:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203984.388520:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203984.388522:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203984.388527:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5439 00000100:00080000:0.0:1788203984.388531:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203984.388535:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203984.388546:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203984.388560:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187899648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203984.388564:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187899776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203984.388578:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203984.388583:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203984.388585:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203984.388589:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5439 00000100:00080000:0.0:1788203984.388593:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203984.388597:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203984.388603:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203984.388616:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187899904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203984.388623:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203984.388629:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1788203984.388629:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203984.388633:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203984.388638:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5439 00000100:00080000:0.0:1788203984.388642:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203984.388645:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203984.388649:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203984.388657:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187900032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203984.388663:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203984.388667:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788203984.388669:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203984.388673:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5439 00000100:00080000:0.0:1788203984.388677:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203984.388682:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203984.388685:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203984/real 1788203984] req@ffff8e828793f100 x1875062187899648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788203984.388690:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203984.388709:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187899648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788203984.388711:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187900160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203984.388759:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203984.388765:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203984.388770:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187899648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203984.388796:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187899776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203984.388823:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203984.388838:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203984/real 1788203984] req@ffff8e828793e680 x1875062187899776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203984.388851:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187899776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203984.388861:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203984.388865:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203984.388868:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187899776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203984.388874:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187899904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203984.388890:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203984.388899:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203984/real 1788203984] req@ffff8e828793f800 x1875062187899904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203984.388907:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187899904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203984.388913:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203984.388915:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203984.388917:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187899904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203984.388920:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187900032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203984.388931:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203984.388936:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203984/real 1788203984] req@ffff8e828793ce00 x1875062187900032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203984.388944:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187900032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203984.388949:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203984.388951:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203984.388953:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187900032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203984.388956:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187900160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203984.388965:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203984.388970:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203984/real 1788203984] req@ffff8e828793c700 x1875062187900160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203984.388976:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187900160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203984.388982:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203984.388984:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203984.388986:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187900160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203985.092518:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203986.116494:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203987.140524:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203988.164498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203989.188508:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203989.508437:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203989.508449:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203989.508454:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203989.508466:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5444 00000100:00080000:0.0:1788203989.508480:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203989.508488:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203989.508529:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187900288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203989.508558:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203989.508567:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203989.508569:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203989.508575:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5444 00000100:00080000:0.0:1788203989.508579:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203989.508584:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203989.508594:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203989.508606:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187900288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203989.508611:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187900416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203989.508626:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203989.508630:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203989.508631:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203989.508651:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5444 00000100:00080000:0.0:1788203989.508656:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203989.508660:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203989.508666:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788203989.508680:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187900544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203989.508691:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203989.508695:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203989.508698:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203989.508702:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5444 00000800:00000100:3.0:1788203989.508708:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788203989.508709:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203989.508717:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203989.508723:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203989.508736:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187900672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203989.508746:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203989.508750:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788203989.508752:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203989/real 1788203989] req@ffff8e828793ca80 x1875062187900288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203989.508753:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203989.508762:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5444 00000100:00080000:0.0:1788203989.508770:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203989.508777:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203989.508781:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187900288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788203989.508783:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788203989.508796:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788203989.508802:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187900800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788203989.508802:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203989.508808:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187900288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203989.508833:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187900416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203989.508860:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203989.508875:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203989/real 1788203989] req@ffff8e828793c000 x1875062187900416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203989.508890:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187900416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203989.508900:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203989.508904:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203989.508908:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187900416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203989.508916:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187900544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203989.508935:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203989.508947:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203989/real 1788203989] req@ffff8e828793dc00 x1875062187900544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203989.508960:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187900544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203989.508969:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203989.508973:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203989.508976:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187900544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203989.508982:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187900672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203989.508998:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203989.509019:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203989/real 1788203989] req@ffff8e828793f480 x1875062187900672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203989.509031:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187900672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203989.509040:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203989.509044:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203989.509047:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187900672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203989.509052:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187900800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203989.509072:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203989.509085:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203989/real 1788203989] req@ffff8e828793c380 x1875062187900800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203989.509098:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187900800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203989.509107:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203989.509110:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203989.509114:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187900800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203990.212469:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203991.236528:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203992.260464:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203993.284485:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203994.308524:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788203994.628484:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203994.628496:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788203994.628500:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203994.628511:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5449 00000100:00080000:0.0:1788203994.628522:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203994.628529:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788203994.628598:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187900928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203994.628627:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203994.628631:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788203994.628633:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203994.628637:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5449 00000100:00080000:0.0:1788203994.628642:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203994.628648:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203994.628659:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788203994.628669:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187900928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788203994.628676:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187901056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203994.628690:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203994.628694:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788203994.628697:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203994.628702:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5449 00000100:00080000:0.0:1788203994.628706:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203994.628710:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203994.628717:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788203994.628726:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788203994.628732:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187901184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788203994.628745:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203994.628748:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788203994.628750:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788203994.628755:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5449 00000100:00080000:0.0:1788203994.628759:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788203994.628764:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788203994.628770:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203994/real 1788203994] req@ffff8e828793e300 x1875062187900928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788203994.628772:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788203994.628792:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187901312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203994.628797:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187900928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788203994.628813:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788203994.628818:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788203994.628819:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788203994.628821:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788203994.628829:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:0.0:1788203994.628830:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5449 00000100:00080000:0.0:1788203994.628837:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788203994.628840:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187900928:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788203994.628844:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788203994.628852:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788203994.628868:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187901056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788203994.628871:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187901440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788203994.628895:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203994.628912:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203994/real 1788203994] req@ffff8e828793d500 x1875062187901056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203994.628929:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187901056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203994.628941:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203994.628943:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203994.628945:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187901056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203994.628949:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187901184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203994.628965:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203994.628972:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203994/real 1788203994] req@ffff8e828793fb80 x1875062187901184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203994.628980:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187901184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203994.628985:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203994.628987:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203994.628989:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187901184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203994.628992:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187901312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203994.629003:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203994.629008:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203994/real 1788203994] req@ffff8e828793d880 x1875062187901312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203994.629015:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187901312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203994.629020:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203994.629022:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203994.629025:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187901312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203994.629035:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187901440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203994.629049:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203994.629060:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203994/real 1788203994] req@ffff8e828793c380 x1875062187901440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203994.629069:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187901440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203994.629075:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203994.629077:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203994.629079:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187901440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788203995.332466:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203996.356474:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203997.380481:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203998.404500:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788203999.428520:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788203999.748432:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203999.748442:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788203999.748446:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203999.748473:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5454 00000100:00080000:1.0:1788203999.748484:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203999.748491:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788203999.748527:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187901568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203999.748554:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203999.748557:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788203999.748559:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203999.748563:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5454 00000100:00080000:1.0:1788203999.748567:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203999.748570:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203999.748579:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203999.748593:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187901696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203999.748606:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203999.748611:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788203999.748612:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187901568:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788203999.748613:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203999.748621:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5454 00000100:00080000:1.0:1788203999.748624:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203999.748627:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203999.748631:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788203999.748641:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187901824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203999.748648:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203999.748651:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788203999.748652:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788203999.748654:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5454 00000100:00080000:1.0:1788203999.748657:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203999.748659:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203999.748663:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203999.748670:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187901952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788203999.748676:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788203999.748678:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788203999.748679:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788203999.748685:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788203999.748687:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5454 00000100:00080000:1.0:1788203999.748692:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788203999.748695:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788203999.748698:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788203999.748707:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187902080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788203999.748740:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203999/real 1788203999] req@ffff8e828d722300 x1875062187901568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203999.748760:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187901568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203999.748772:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203999.748775:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203999.748778:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187901568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788203999.748808:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187901696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203999.748827:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203999.748836:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203999/real 1788203999] req@ffff8e828d722d80 x1875062187901696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203999.748844:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187901696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203999.748850:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203999.748852:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203999.748854:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187901696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203999.748858:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187901824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788203999.748869:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203999.748875:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203999/real 1788203999] req@ffff8e828d721180 x1875062187901824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203999.748882:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187901824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203999.748887:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203999.748889:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203999.748891:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187901824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788203999.748894:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187901952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203999.748905:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203999.748910:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203999/real 1788203999] req@ffff8e828d720380 x1875062187901952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203999.748917:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187901952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203999.748923:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203999.748925:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203999.748927:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187901952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788203999.748930:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187902080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788203999.748939:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788203999.748944:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788203999/real 1788203999] req@ffff8e828d720a80 x1875062187902080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788203999.748951:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187902080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788203999.748956:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788203999.748958:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788203999.748960:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187902080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204000.452508:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204001.476439:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204002.500486:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204003.524481:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204004.548482:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204004.868443:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204004.868455:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204004.868460:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204004.868473:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5459 00000100:00080000:1.0:1788204004.868484:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204004.868493:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204004.868554:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187902208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204004.868597:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204004.868602:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204004.868605:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204004.868611:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5459 00000100:00080000:1.0:1788204004.868616:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204004.868622:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204004.868633:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204004.868650:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187902208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204004.868656:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187902336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204004.868691:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204004.868696:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204004.868698:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788204004.868703:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788204004.868707:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5459 00000100:00080000:1.0:1788204004.868714:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204004.868720:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204004.868728:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204004.868739:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204004/real 1788204004] req@ffff8e828d720a80 x1875062187902208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204004.868749:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187902464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204004.868756:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187902208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204004.868774:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204004.868777:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204004.868779:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:1.0:1788204004.868785:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788204004.868786:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187902208:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204004.868791:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204004.868799:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5459 00000100:00080000:1.0:1788204004.868804:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788204004.868807:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187902336:192.168.201.121@tcp:38:kworker.0 00000100:00080000:1.0:1788204004.868812:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204004.868820:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788204004.868825:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204004.868837:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204004/real 1788204004] req@ffff8e828d720380 x1875062187902336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204004.868839:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187902592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204004.868849:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187902336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204004.868866:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204004.868871:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204004.868873:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204004.868874:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204004.868880:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00080000:1.0:1788204004.868883:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5459 00000100:00100000:2.0:1788204004.868885:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187902336:192.168.201.121@tcp:38:kworker.0 00000100:00080000:1.0:1788204004.868893:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788204004.868893:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187902464:192.168.201.121@tcp:38:kworker.0 00000100:00080000:1.0:1788204004.868905:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204004.868913:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788204004.868914:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204004.868927:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204004/real 1788204004] req@ffff8e828d721180 x1875062187902464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204004.868935:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187902720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204004.868939:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187902464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204004.868966:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204004.868968:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204004.868971:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187902464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204004.868981:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187902592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204004.868995:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204004.869002:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204004/real 1788204004] req@ffff8e828d722d80 x1875062187902592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204004.869012:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187902592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204004.869017:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204004.869018:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204004.869020:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187902592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204004.869023:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187902720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204004.869031:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204004.869036:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204004/real 1788204004] req@ffff8e828d722300 x1875062187902720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204004.869042:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187902720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204004.869046:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204004.869047:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204004.869049:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187902720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204005.572517:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204006.596495:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204007.620479:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204008.644477:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204009.668472:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204009.988443:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204009.988452:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204009.988456:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204009.988463:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5465 00000100:00080000:1.0:1788204009.988471:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204009.988476:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204009.988506:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187902848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204009.988562:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187902848:192.168.201.121@tcp:250:kworker.0 00000800:00000100:3.0:1788204009.988619:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788204009.988649:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204009.988653:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204009.988655:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204009.988664:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5465 00000100:00080000:1.0:1788204009.988668:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204009.988672:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204009.988680:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204009.988692:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187902976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204009.988737:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204009/real 1788204009] req@ffff8e828d720e00 x1875062187902848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204009.988748:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187902848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204009.988754:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204009.988757:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204009.988760:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187902848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204009.988778:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187902976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204009.988793:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204009.988801:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204009/real 1788204009] req@ffff8e828d723800 x1875062187902976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204009.988808:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187902976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204009.988813:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204009.988815:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204009.988817:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187902976:192.168.201.121@tcp:38:kworker.0 00000100:00080000:1.0:1788204009.988854:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204009.988857:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204009.988859:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204009.988861:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5465 00000100:00080000:1.0:1788204009.988864:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204009.988866:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204009.988870:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204009.988879:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187903104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204009.988884:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204009.988886:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204009.988887:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204009.988890:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5465 00000100:00080000:1.0:1788204009.988893:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204009.988896:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204009.988900:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204009.988910:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187903232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204009.988918:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204009.988921:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204009.988923:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204009.988926:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5465 00000100:00080000:1.0:1788204009.988929:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204009.988932:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204009.988936:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204009.988945:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187903360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204009.989003:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187903104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204009.989041:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204009.989055:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204009/real 1788204009] req@ffff8e828d721500 x1875062187903104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204009.989069:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187903104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204009.989078:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204009.989082:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204009.989086:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187903104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204009.989093:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187903232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204009.989118:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204009.989128:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204009/real 1788204009] req@ffff8e828d720000 x1875062187903232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204009.989138:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187903232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204009.989144:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204009.989147:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204009.989149:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187903232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204009.989154:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187903360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204009.989170:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204009.989180:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204009/real 1788204009] req@ffff8e828d722680 x1875062187903360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204009.989191:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187903360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204009.989198:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204009.989200:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204009.989208:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187903360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204010.692528:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204011.716517:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204012.740517:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204013.764466:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204014.788503:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204015.108469:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204015.108490:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204015.108495:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204015.108508:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5470 00000100:00080000:0.0:1788204015.108523:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204015.108537:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204015.108597:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187903488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204015.108633:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204015.108637:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204015.108639:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204015.108644:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5470 00000100:00080000:0.0:1788204015.108648:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204015.108653:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204015.108664:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204015.108678:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187903616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204015.108686:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187903488:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204015.108691:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204015.108696:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204015.108698:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204015.108702:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5470 00000100:00080000:0.0:1788204015.108707:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204015.108712:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204015.108717:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204015.108728:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187903744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204015.108736:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204015.108738:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204015.108740:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204015.108742:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5470 00000100:00080000:0.0:1788204015.108745:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204015.108748:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204015.108751:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204015.108758:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187903872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204015.108764:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204015.108766:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204015.108768:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204015.108772:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5470 00000100:00080000:0.0:1788204015.108774:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204015.108776:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204015.108778:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788204015.108780:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204015.108784:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187904000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204015.108839:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204015/real 1788204015] req@ffff8e828793f480 x1875062187903488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204015.108858:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187903488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204015.108876:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204015.108881:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204015.108885:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187903488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204015.108910:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187903616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204015.108933:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204015.108945:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204015/real 1788204015] req@ffff8e828793dc00 x1875062187903616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204015.108956:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187903616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204015.108964:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204015.108967:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204015.108970:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187903616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204015.108976:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187903744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204015.109003:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204015.109013:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204015/real 1788204015] req@ffff8e828793c000 x1875062187903744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204015.109023:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187903744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204015.109031:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204015.109034:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204015.109038:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187903744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204015.109044:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187903872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204015.109062:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204015.109072:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204015/real 1788204015] req@ffff8e828793ca80 x1875062187903872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204015.109082:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187903872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204015.109090:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204015.109093:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204015.109096:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187903872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204015.109102:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187904000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204015.109119:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204015.109128:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204015/real 1788204015] req@ffff8e828793c700 x1875062187904000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204015.109138:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187904000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204015.109146:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204015.109148:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204015.109151:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187904000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204015.812539:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204016.836508:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204017.860537:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204018.884503:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204019.908508:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204020.228426:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204020.228436:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204020.228441:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204020.228452:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5475 00000100:00080000:0.0:1788204020.228464:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204020.228472:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204020.228511:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187904128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204020.228532:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204020.228535:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204020.228536:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204020.228539:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5475 00000100:00080000:0.0:1788204020.228542:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204020.228544:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204020.228552:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204020.228560:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187904256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204020.228567:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204020.228569:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204020.228571:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204020.228574:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5475 00000100:00080000:0.0:1788204020.228576:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204020.228578:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204020.228581:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204020.228587:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187904384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204020.228593:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204020.228596:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788204020.228596:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187904128:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204020.228598:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204020.228605:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5475 00000100:00080000:0.0:1788204020.228607:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204020.228609:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204020.228612:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204020.228618:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187904512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204020.228623:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204020.228625:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204020.228627:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204020.228629:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5475 00000100:00080000:0.0:1788204020.228631:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204020.228633:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204020.228636:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204020.228644:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187904640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204020.228682:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204020.228740:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204020/real 1788204020] req@ffff8e828793ce00 x1875062187904128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204020.228761:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187904128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204020.228773:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204020.228777:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204020.228782:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187904128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204020.228790:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187904256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204020.228811:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204020.228823:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204020/real 1788204020] req@ffff8e828793f800 x1875062187904256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204020.228835:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187904256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204020.228844:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204020.228848:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204020.228851:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187904256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204020.228878:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187904384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204020.228895:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204020.228906:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204020/real 1788204020] req@ffff8e828793e680 x1875062187904384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204020.228918:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187904384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204020.228932:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204020.228935:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204020.228938:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187904384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204020.228944:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187904512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204020.228961:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204020.228970:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204020/real 1788204020] req@ffff8e828793f100 x1875062187904512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204020.228981:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187904512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204020.228989:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204020.228992:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204020.228996:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187904512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204020.229002:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187904640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204020.229018:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204020.229028:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204020/real 1788204020] req@ffff8e828793c700 x1875062187904640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204020.229039:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187904640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204020.229046:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204020.229049:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204020.229053:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187904640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204020.932507:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204021.956510:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204022.980507:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204024.004503:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204025.028491:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204025.348475:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204025.348488:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204025.348492:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204025.348504:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5480 00000100:00080000:2.0:1788204025.348522:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204025.348534:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204025.348587:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187904768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204025.348616:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204025.348619:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204025.348620:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204025.348624:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5480 00000100:00080000:2.0:1788204025.348626:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204025.348629:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204025.348639:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204025.348650:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187904896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204025.348657:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204025.348660:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204025.348661:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204025.348663:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5480 00000100:00080000:2.0:1788204025.348665:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204025.348667:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204025.348670:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204025.348675:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187904768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204025.348679:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187905024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204025.348688:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204025.348690:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204025.348691:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204025.348693:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5480 00000100:00080000:2.0:1788204025.348696:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204025.348698:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204025.348701:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204025.348708:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187905152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204025.348714:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204025.348715:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204025.348716:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204025.348718:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5480 00000100:00080000:2.0:1788204025.348720:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204025.348722:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204025.348724:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204025.348732:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187905280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204025.348768:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204025.348842:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204025/real 1788204025] req@ffff8e82894e7800 x1875062187904768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204025.348865:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187904768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204025.348877:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204025.348881:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204025.348886:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187904768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204025.348894:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187904896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204025.348917:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204025.348930:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204025/real 1788204025] req@ffff8e82894e6300 x1875062187904896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204025.348942:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187904896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204025.348951:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204025.348954:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204025.348958:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187904896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204025.348987:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187905024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204025.349007:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204025.349033:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204025/real 1788204025] req@ffff8e82894e4a80 x1875062187905024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204025.349045:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187905024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204025.349052:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204025.349055:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204025.349059:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187905024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204025.349065:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187905152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204025.349083:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204025.349093:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204025/real 1788204025] req@ffff8e82894e4e00 x1875062187905152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204025.349104:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187905152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204025.349112:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204025.349115:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204025.349118:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187905152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204025.349123:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187905280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204025.349140:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204025.349150:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204025/real 1788204025] req@ffff8e82894e6d80 x1875062187905280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204025.349159:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187905280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204025.349166:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204025.349169:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204025.349172:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187905280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204026.052528:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204027.076507:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204028.100497:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204029.124484:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204030.148516:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204030.468389:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204030.468400:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204030.468405:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204030.468416:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5485 00000100:00080000:2.0:1788204030.468428:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204030.468435:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204030.468476:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187905408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204030.468502:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204030.468508:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204030.468511:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204030.468516:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5485 00000100:00080000:2.0:1788204030.468521:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204030.468526:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204030.468537:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204030.468539:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187905408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204030.468550:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187905536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204030.468560:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204030.468562:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204030.468564:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204030.468566:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5485 00000100:00080000:2.0:1788204030.468569:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204030.468571:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204030.468575:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204030.468582:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187905664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204030.468588:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204030.468590:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204030.468592:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204030.468594:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5485 00000100:00080000:2.0:1788204030.468596:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204030.468599:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204030.468603:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204030.468612:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187905792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204030.468612:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204030.468621:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204030.468624:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204030.468625:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204030.468628:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5485 00000100:00080000:2.0:1788204030.468630:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204030.468632:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204030.468636:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204030.468643:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187905920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204030.468660:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204030/real 1788204030] req@ffff8e82894e7100 x1875062187905408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204030.468673:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187905408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204030.468681:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204030.468684:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204030.468687:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187905408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204030.468707:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187905536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204030.468720:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204030.468728:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204030/real 1788204030] req@ffff8e82894e5c00 x1875062187905536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204030.468735:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187905536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204030.468740:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204030.468742:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204030.468744:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187905536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204030.468748:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187905664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204030.468758:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204030.468764:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204030/real 1788204030] req@ffff8e82894e4380 x1875062187905664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204030.468770:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187905664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204030.468774:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204030.468776:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204030.468778:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187905664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204030.468781:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187905792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204030.468791:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204030.468796:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204030/real 1788204030] req@ffff8e82894e5f80 x1875062187905792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204030.468802:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187905792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204030.468806:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204030.468808:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204030.468810:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187905792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204030.468812:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187905920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204030.468821:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204030.468826:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204030/real 1788204030] req@ffff8e82894e6d80 x1875062187905920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204030.468833:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187905920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204030.468838:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204030.468839:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204030.468841:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187905920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204031.172487:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204032.196488:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204033.220509:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204034.244495:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204035.268494:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204035.588446:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204035.588458:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204035.588463:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204035.588476:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5490 00000100:00080000:1.0:1788204035.588488:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204035.588496:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204035.588552:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187906048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204035.588576:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204035.588579:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204035.588580:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204035.588583:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5490 00000100:00080000:1.0:1788204035.588586:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204035.588589:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204035.588599:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204035.588618:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187906176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204035.588629:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204035.588634:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788204035.588634:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187906048:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204035.588637:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204035.588645:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5490 00000100:00080000:1.0:1788204035.588648:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204035.588650:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204035.588654:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204035.588664:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187906304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204035.588670:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204035.588672:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204035.588673:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204035.588676:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5490 00000100:00080000:1.0:1788204035.588679:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204035.588681:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204035.588684:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204035.588690:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187906432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204035.588695:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204035.588697:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204035.588699:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204035.588701:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5490 00000100:00080000:1.0:1788204035.588704:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204035.588706:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204035.588709:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204035.588714:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187906560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204035.588779:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187906176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204035.588812:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187906304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204035.588849:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187906432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204035.588873:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187906560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204036.292480:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204037.316531:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204038.340429:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788204038.660638:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:2.0:1788204038.660697:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788204038.660717:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788204038.660766:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788204038.660777:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788204038.661018:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788204038.661028:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:3.0:1788204038.661117:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204035/real 1788204038] req@ffff8e828d720000 x1875062187906176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204038.661147:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187906176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204038.661160:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204038.661165:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204038.661170:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187906176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204038.661178:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204035/real 1788204038] req@ffff8e828d721500 x1875062187906304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204038.661194:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187906304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204038.661201:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204038.661204:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204038.661286:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187906304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204038.661294:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204035/real 1788204038] req@ffff8e828d723800 x1875062187906432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204038.661341:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187906432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204038.661350:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204038.661354:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204038.661358:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187906432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204038.661365:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204035/real 1788204038] req@ffff8e828d720e00 x1875062187906560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204038.661383:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187906560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204038.661391:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204038.661394:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204038.661397:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187906560:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204038.661405:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204035/real 1788204038] req@ffff8e828d722680 x1875062187906048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204038.661421:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187906048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204038.661429:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204038.661432:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204038.661436:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187906048:192.168.201.121@tcp:250:kworker.0 00010000:00010000:0.0:1788204039.364444:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204040.388481:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204040.708457:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204040.708469:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204040.708473:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204040.708485:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5495 00000100:00080000:2.0:1788204040.708496:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204040.708505:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204040.708541:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187906688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204040.708567:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204040.708570:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204040.708571:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204040.708576:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5495 00000100:00080000:2.0:1788204040.708578:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204040.708582:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204040.708591:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204040.708601:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187906688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204040.708607:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187906816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204040.708619:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204040.708622:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204040.708624:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204040.708628:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5495 00000100:00080000:2.0:1788204040.708631:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204040.708635:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204040.708640:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204040.708653:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187906944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204040.708662:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204040.708666:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1788204040.708678:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204040.708683:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204040.708688:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5495 00000100:00080000:2.0:1788204040.708693:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204040.708698:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204040.708704:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204040.708718:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187907072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204040.708727:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204040.708731:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788204040.708734:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204040/real 1788204040] req@ffff8e82894e4e00 x1875062187906688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204040.708735:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204040.708743:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5495 00000100:00080000:2.0:1788204040.708767:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204040.708773:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204040.708779:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204040.708781:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187906688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204040.708796:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1788204040.708797:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187907200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204040.708804:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204040.708818:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187906688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204040.708851:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187906816:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204040.708879:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204040.708895:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204040/real 1788204040] req@ffff8e82894e4a80 x1875062187906816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204040.708908:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187906816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204040.708917:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204040.708921:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204040.708924:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187906816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204040.708929:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187906944:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204040.708952:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204040.708963:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204040/real 1788204040] req@ffff8e82894e6300 x1875062187906944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204040.708974:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187906944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204040.708982:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204040.708986:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204040.708989:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187906944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204040.708994:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187907072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204040.709020:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204040.709034:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204040/real 1788204040] req@ffff8e82894e7800 x1875062187907072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204040.709045:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187907072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204040.709052:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204040.709056:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204040.709059:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187907072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204040.709065:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187907200:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204040.709086:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204040.709095:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204040/real 1788204040] req@ffff8e82894e4000 x1875062187907200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204040.709105:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187907200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204040.709112:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204040.709115:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204040.709118:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187907200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204041.412484:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204042.436446:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204043.460446:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204044.484470:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204045.508488:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204045.828420:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204045.828431:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204045.828435:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204045.828445:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5500 00000100:00080000:1.0:1788204045.828457:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204045.828464:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204045.828505:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187907328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204045.828527:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204045.828529:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204045.828531:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204045.828534:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5500 00000100:00080000:1.0:1788204045.828536:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204045.828539:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204045.828548:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204045.828557:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187907456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204045.828565:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204045.828569:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788204045.828571:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187907328:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204045.828571:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204045.828576:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5500 00000100:00080000:1.0:1788204045.828582:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204045.828584:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204045.828587:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204045.828593:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187907584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204045.828599:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204045.828601:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204045.828603:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204045.828605:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5500 00000100:00080000:1.0:1788204045.828607:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204045.828609:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204045.828612:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204045.828619:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187907712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204045.828624:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204045.828626:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204045.828627:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204045.828629:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5500 00000100:00080000:1.0:1788204045.828643:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204045.828647:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204045.828652:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788204045.828654:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204045.828679:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187907840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204045.828717:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204045/real 1788204045] req@ffff8e828d722300 x1875062187907328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204045.828734:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187907328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204045.828742:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204045.828745:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204045.828748:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187907328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204045.828777:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187907456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204045.828792:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204045.828801:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204045/real 1788204045] req@ffff8e828d722d80 x1875062187907456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204045.828808:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187907456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204045.828814:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204045.828816:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204045.828818:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187907456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204045.828821:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187907584:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204045.828831:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204045.828837:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204045/real 1788204045] req@ffff8e828d721180 x1875062187907584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204045.828844:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187907584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204045.828853:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204045.828854:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204045.828856:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187907584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204045.828859:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187907712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204045.828869:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204045.828874:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204045/real 1788204045] req@ffff8e828d720380 x1875062187907712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204045.828881:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187907712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204045.828893:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204045.828895:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204045.828897:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187907712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204045.828901:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187907840:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204045.828915:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204045.828924:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204045/real 1788204045] req@ffff8e828d720a80 x1875062187907840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204045.828935:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187907840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204045.828940:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204045.828942:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204045.828944:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187907840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204046.532568:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204047.556529:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204048.580506:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204049.604522:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204050.628475:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204050.948430:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204050.948440:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204050.948445:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204050.948458:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5506 00000100:00080000:0.0:1788204050.948469:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204050.948476:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204050.948515:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187907968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204050.948546:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204050.948550:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204050.948552:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204050.948557:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5506 00000100:00080000:0.0:1788204050.948560:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204050.948564:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204050.948587:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204050.948606:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187908096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204050.948606:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187907968:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204050.948620:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204050.948622:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204050.948624:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204050.948627:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5506 00000100:00080000:0.0:1788204050.948630:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204050.948633:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204050.948638:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204050.948647:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187908224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204050.948654:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204050.948657:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204050.948659:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204050.948663:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5506 00000100:00080000:0.0:1788204050.948666:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204050.948670:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204050.948675:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788204050.948684:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204050.948689:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187908352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204050.948699:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204050.948703:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204050.948704:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204050.948708:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5506 00000100:00080000:0.0:1788204050.948710:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204050.948713:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204050.948717:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204050.948727:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187908480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204050.948735:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204050/real 1788204050] req@ffff8e828793ca80 x1875062187907968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204050.948765:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187907968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204050.948776:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204050.948780:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204050.948783:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187907968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204050.948805:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187908096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204050.948822:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204050.948831:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204050/real 1788204050] req@ffff8e828793c000 x1875062187908096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204050.948838:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187908096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204050.948843:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204050.948846:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204050.948848:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187908096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204050.948852:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187908224:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204050.948862:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204050.948867:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204050/real 1788204050] req@ffff8e828793dc00 x1875062187908224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204050.948874:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187908224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204050.948879:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204050.948880:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204050.948882:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187908224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204050.948885:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187908352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204050.948903:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204050.948909:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204050/real 1788204050] req@ffff8e828793f480 x1875062187908352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204050.948915:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187908352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204050.948920:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204050.948922:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204050.948923:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187908352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204050.948927:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187908480:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204050.948945:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204050.948951:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204050/real 1788204050] req@ffff8e828793c380 x1875062187908480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204050.948958:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187908480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204050.948962:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204050.948963:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204050.948965:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187908480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204051.652520:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204052.676531:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204053.700510:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204054.724479:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204055.748515:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204056.068474:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204056.068499:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204056.068504:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204056.068516:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5511 00000100:00080000:1.0:1788204056.068530:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204056.068539:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204056.068588:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187908608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204056.068616:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204056.068621:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204056.068623:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204056.068627:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5511 00000100:00080000:1.0:1788204056.068631:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204056.068635:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204056.068646:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204056.068660:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187908608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204056.068661:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187908736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204056.068675:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204056.068678:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204056.068679:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204056.068683:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5511 00000100:00080000:1.0:1788204056.068686:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204056.068690:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204056.068694:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204056.068704:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187908864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204056.068713:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204056.068717:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204056.068719:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204056.068723:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5511 00000100:00080000:1.0:1788204056.068726:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204056.068729:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204056.068734:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788204056.068734:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204056.068746:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187908992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204056.068755:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204056.068758:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204056.068760:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204056.068764:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5511 00000100:00080000:1.0:1788204056.068767:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204056.068770:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204056.068774:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204056.068783:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187909120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204056.068787:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204056/real 1788204056] req@ffff8e828d720a80 x1875062187908608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204056.068810:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187908608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204056.068819:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204056.068822:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204056.068826:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187908608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204056.068846:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187908736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204056.068861:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204056.068868:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204056/real 1788204056] req@ffff8e828d720380 x1875062187908736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204056.068880:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187908736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204056.068885:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204056.068887:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204056.068889:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187908736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204056.068892:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187908864:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204056.068902:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204056.068907:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204056/real 1788204056] req@ffff8e828d721180 x1875062187908864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204056.068914:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187908864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204056.068918:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204056.068920:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204056.068921:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187908864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204056.068924:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187908992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204056.068933:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204056.068938:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204056/real 1788204056] req@ffff8e828d722d80 x1875062187908992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204056.068945:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187908992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204056.068953:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204056.068955:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204056.068957:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187908992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204056.068959:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187909120:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204056.068968:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204056.068973:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204056/real 1788204056] req@ffff8e828d722300 x1875062187909120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204056.068978:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187909120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204056.068982:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204056.068983:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204056.068985:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187909120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204056.772514:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204057.796528:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204058.820585:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204059.844515:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204060.868545:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204061.188472:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204061.188480:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204061.188483:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204061.188491:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5516 00000100:00080000:0.0:1788204061.188499:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204061.188504:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204061.188533:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187909248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204061.188554:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204061.188557:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204061.188558:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204061.188561:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5516 00000100:00080000:0.0:1788204061.188562:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204061.188565:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204061.188570:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204061.188578:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187909376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204061.188584:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204061.188586:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204061.188587:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204061.188590:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5516 00000100:00080000:0.0:1788204061.188593:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204061.188595:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204061.188598:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204061.188606:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187909248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204061.188609:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187909504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204061.188622:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204061.188625:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204061.188627:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204061.188631:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5516 00000100:00080000:0.0:1788204061.188639:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204061.188644:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204061.188649:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204061.188659:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187909632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204061.188666:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204061.188672:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204061.188675:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204061.188677:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204061.188681:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5516 00000100:00080000:0.0:1788204061.188685:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204061.188687:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204061.188690:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204061.188700:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187909760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204061.188704:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204061/real 1788204061] req@ffff8e828793d880 x1875062187909248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204061.188726:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187909248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204061.188737:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204061.188741:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204061.188745:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187909248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204061.188753:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187909376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204061.188775:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204061.188787:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204061/real 1788204061] req@ffff8e828793fb80 x1875062187909376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204061.188801:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187909376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204061.188806:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204061.188808:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204061.188811:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187909376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204061.188828:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187909504:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204061.188842:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204061.188848:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204061/real 1788204061] req@ffff8e828793d500 x1875062187909504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204061.188855:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187909504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204061.188865:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204061.188867:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204061.188869:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187909504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204061.188872:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187909632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204061.188883:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204061.188889:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204061/real 1788204061] req@ffff8e828793e300 x1875062187909632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204061.188895:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187909632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204061.188900:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204061.188901:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204061.188903:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187909632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204061.188906:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187909760:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204061.188915:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204061.188920:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204061/real 1788204061] req@ffff8e828793c380 x1875062187909760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204061.188925:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187909760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204061.188930:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204061.188932:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204061.188933:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187909760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204061.892520:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204062.916501:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204063.940512:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204064.964530:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204065.988637:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204066.308415:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204066.308425:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204066.308430:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204066.308441:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5521 00000100:00080000:0.0:1788204066.308453:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204066.308460:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204066.308509:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187909888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204066.308545:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204066.308550:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204066.308552:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204066.308557:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5521 00000100:00080000:0.0:1788204066.308561:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204066.308566:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204066.308578:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204066.308593:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187910016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204066.308601:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187909888:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204066.308610:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204066.308614:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204066.308616:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204066.308621:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5521 00000100:00080000:0.0:1788204066.308635:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204066.308641:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204066.308647:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204066.308661:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187910144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204066.308670:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204066.308674:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204066.308678:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204066.308681:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204066.308687:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5521 00000100:00080000:0.0:1788204066.308692:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204066.308697:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204066.308703:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204066.308715:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204066/real 1788204066] req@ffff8e828793f480 x1875062187909888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204066.308719:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187910272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204066.308735:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204066.308738:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187909888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204066.308740:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204066.308743:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204066.308751:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5521 00000100:00080000:0.0:1788204066.308757:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204066.308762:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:3.0:1788204066.308763:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788204066.308767:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788204066.308769:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204066.308774:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187909888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204066.308781:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187910400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204066.308800:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187910016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204066.308825:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204066.308838:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204066/real 1788204066] req@ffff8e828793dc00 x1875062187910016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204066.308860:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187910016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204066.308866:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204066.308868:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204066.308870:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187910016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204066.308875:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187910144:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204066.308889:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204066.308896:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204066/real 1788204066] req@ffff8e828793c000 x1875062187910144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204066.308903:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187910144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204066.308908:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204066.308910:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204066.308912:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187910144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204066.308916:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187910272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204066.308933:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204066.308939:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204066/real 1788204066] req@ffff8e828793ca80 x1875062187910272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204066.308946:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187910272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204066.308951:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204066.308952:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204066.308954:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187910272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204066.308957:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187910400:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204066.308966:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204066.308971:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204066/real 1788204066] req@ffff8e828793c700 x1875062187910400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204066.308977:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187910400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204066.308982:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204066.308983:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204066.308984:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187910400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204067.012494:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204068.036459:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204069.060520:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204070.084479:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204071.108502:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204071.428478:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204071.428489:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204071.428494:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204071.428506:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5526 00000100:00080000:0.0:1788204071.428517:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204071.428524:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204071.428565:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187910528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204071.428591:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204071.428594:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204071.428596:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204071.428599:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5526 00000100:00080000:0.0:1788204071.428602:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204071.428605:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204071.428615:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204071.428623:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187910656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204071.428631:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204071.428634:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204071.428637:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788204071.428638:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187910528:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204071.428641:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5526 00000100:00080000:0.0:1788204071.428646:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204071.428649:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204071.428652:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204071.428660:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187910784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204071.428666:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204071.428668:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204071.428669:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204071.428672:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5526 00000100:00080000:0.0:1788204071.428674:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204071.428676:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204071.428679:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204071.428685:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187910912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204071.428691:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204071.428693:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204071.428693:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204071.428696:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5526 00000100:00080000:0.0:1788204071.428699:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204071.428702:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204071.428705:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204071.428713:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187911040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204071.428717:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204071.428770:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204071/real 1788204071] req@ffff8e828793f100 x1875062187910528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204071.428784:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187910528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204071.428793:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204071.428795:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204071.428799:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187910528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204071.428818:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187910656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204071.428832:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204071.428840:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204071/real 1788204071] req@ffff8e828793e680 x1875062187910656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204071.428847:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187910656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204071.428852:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204071.428854:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204071.428856:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187910656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204071.428859:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187910784:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204071.428868:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204071.428873:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204071/real 1788204071] req@ffff8e828793f800 x1875062187910784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204071.428879:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187910784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204071.428888:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204071.428890:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204071.428892:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187910784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204071.428895:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187910912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204071.428904:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204071.428909:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204071/real 1788204071] req@ffff8e828793ce00 x1875062187910912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204071.428915:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187910912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204071.428920:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204071.428921:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204071.428923:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187910912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204071.428926:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187911040:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204071.428934:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204071.428939:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204071/real 1788204071] req@ffff8e828793c700 x1875062187911040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204071.428944:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187911040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204071.428948:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204071.428950:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204071.428952:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187911040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204072.132503:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204073.156518:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204074.180526:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204075.204454:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204076.228486:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204076.548434:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204076.548440:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204076.548447:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204076.548456:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5531 00000100:00080000:0.0:1788204076.548464:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204076.548470:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204076.548498:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187911168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204076.548516:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204076.548518:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204076.548520:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204076.548522:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5531 00000100:00080000:0.0:1788204076.548524:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204076.548527:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204076.548533:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204076.548542:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187911296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204076.548549:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204076.548554:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204076.548555:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788204076.548555:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187911168:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204076.548563:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5531 00000100:00080000:0.0:1788204076.548566:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204076.548568:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204076.548572:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204076.548581:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187911424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204076.548586:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204076.548588:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204076.548590:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204076.548593:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5531 00000100:00080000:0.0:1788204076.548596:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204076.548598:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204076.548601:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204076.548606:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187911552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204076.548611:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204076.548613:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204076.548615:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204076.548617:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5531 00000100:00080000:0.0:1788204076.548620:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788204076.548625:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204076.548626:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204076.548631:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204076.548639:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187911680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204076.548669:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204076/real 1788204076] req@ffff8e828793ca80 x1875062187911168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204076.548687:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187911168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204076.548697:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204076.548702:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204076.548706:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187911168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204076.548738:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187911296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204076.548755:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204076.548764:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204076/real 1788204076] req@ffff8e828793c000 x1875062187911296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204076.548773:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187911296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204076.548778:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204076.548780:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204076.548782:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187911296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204076.548786:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187911424:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204076.548796:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204076.548803:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204076/real 1788204076] req@ffff8e828793dc00 x1875062187911424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204076.548809:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187911424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204076.548813:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204076.548815:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204076.548817:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187911424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204076.548820:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187911552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204076.548832:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204076.548838:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204076/real 1788204076] req@ffff8e828793f480 x1875062187911552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204076.548845:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187911552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204076.548850:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204076.548852:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204076.548854:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187911552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204076.548857:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187911680:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204076.548868:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204076.548873:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204076/real 1788204076] req@ffff8e828793c380 x1875062187911680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204076.548883:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187911680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204076.548888:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204076.548890:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204076.548892:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187911680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204077.252444:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204078.276469:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204079.300489:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204080.324474:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204081.348464:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204081.668410:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204081.668419:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204081.668423:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204081.668431:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5536 00000100:00080000:2.0:1788204081.668442:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204081.668450:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204081.668484:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187911808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204081.668519:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204081.668523:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204081.668528:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204081.668534:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5536 00000100:00080000:2.0:1788204081.668538:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204081.668543:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187911808:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204081.668546:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204081.668559:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204081.668575:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187911936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204081.668586:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204081.668589:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204081.668591:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204081.668594:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5536 00000100:00080000:2.0:1788204081.668598:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788204081.668606:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204081.668615:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204081.668621:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204081.668634:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187912064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204081.668644:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204081.668645:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204081/real 1788204081] req@ffff8e82894e6680 x1875062187911808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204081.668649:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204081.668653:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204081.668662:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5536 00000100:00080000:2.0:1788204081.668667:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204081.668668:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187911808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204081.668675:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:3.0:1788204081.668681:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1788204081.668684:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788204081.668688:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204081.668693:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187911808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204081.668701:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187912192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204081.668714:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204081.668716:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187911936:192.168.201.121@tcp:38:kworker.0 00000100:00080000:2.0:1788204081.668718:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204081.668722:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204081.668726:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5536 00000100:00080000:2.0:1788204081.668729:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204081.668733:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204081.668737:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788204081.668742:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204081.668748:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187912320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204081.668757:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204081/real 1788204081] req@ffff8e82894e5880 x1875062187911936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204081.668772:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187911936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204081.668780:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204081.668782:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204081.668785:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187911936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204081.668791:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187912064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204081.668808:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204081.668818:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204081/real 1788204081] req@ffff8e82894e6a00 x1875062187912064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204081.668829:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187912064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204081.668835:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204081.668838:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204081.668841:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187912064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204081.668853:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187912192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204081.668872:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204081.668882:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204081/real 1788204081] req@ffff8e82894e7480 x1875062187912192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204081.668893:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187912192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204081.668899:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204081.668901:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204081.668904:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187912192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204081.668909:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187912320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204081.668925:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204081.668934:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204081/real 1788204081] req@ffff8e82894e6680 x1875062187912320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204081.668944:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187912320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204081.668951:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204081.668954:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204081.668957:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187912320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204082.372437:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204083.396458:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204084.420478:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204085.444541:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204086.468471:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204086.788393:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204086.788400:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204086.788403:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204086.788410:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5541 00000100:00080000:1.0:1788204086.788419:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204086.788424:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204086.788453:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187912448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204086.788475:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204086.788477:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204086.788478:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204086.788481:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5541 00000100:00080000:1.0:1788204086.788483:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204086.788485:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204086.788492:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204086.788500:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187912576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204086.788506:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204086.788508:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204086.788509:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204086.788511:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5541 00000100:00080000:1.0:1788204086.788513:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204086.788515:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204086.788518:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204086.788521:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187912448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204086.788526:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187912704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204086.788545:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204086.788547:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204086.788549:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204086.788551:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5541 00000100:00080000:1.0:1788204086.788553:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204086.788556:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204086.788558:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204086.788564:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187912832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204086.788570:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204086.788572:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204086.788573:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204086.788576:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5541 00000100:00080000:1.0:1788204086.788580:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204086.788583:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204086.788587:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788204086.788590:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204086.788602:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187912960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204086.788637:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204086/real 1788204086] req@ffff8e828d722680 x1875062187912448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204086.788656:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187912448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204086.788665:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204086.788667:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204086.788670:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187912448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204086.788675:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187912576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204086.788691:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204086.788699:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204086/real 1788204086] req@ffff8e828d720e00 x1875062187912576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204086.788707:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187912576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204086.788715:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204086.788718:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204086.788721:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187912576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204086.788744:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187912704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204086.788763:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204086.788772:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204086/real 1788204086] req@ffff8e828d723800 x1875062187912704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204086.788779:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187912704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204086.788784:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204086.788786:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204086.788788:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187912704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204086.788791:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187912832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204086.788802:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204086.788807:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204086/real 1788204086] req@ffff8e828d721500 x1875062187912832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204086.788813:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187912832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204086.788818:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204086.788819:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204086.788821:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187912832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204086.788824:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187912960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204086.788832:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204086.788837:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204086/real 1788204086] req@ffff8e828d720000 x1875062187912960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204086.788843:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187912960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204086.788848:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204086.788849:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204086.788850:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187912960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204087.492473:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204088.516486:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204089.540425:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204090.564504:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204091.588508:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204091.908404:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204091.908412:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204091.908416:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204091.908424:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5546 00000100:00080000:1.0:1788204091.908434:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204091.908439:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204091.908475:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187913088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204091.908503:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204091.908505:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204091.908507:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204091.908510:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5546 00000100:00080000:1.0:1788204091.908513:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204091.908516:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204091.908523:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204091.908532:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187913216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204091.908539:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204091.908542:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204091.908543:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204091.908547:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5546 00000100:00100000:3.0:1788204091.908548:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187913088:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204091.908549:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204091.908555:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204091.908560:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204091.908567:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187913344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204091.908576:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204091.908578:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204091.908580:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204091.908582:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5546 00000100:00080000:1.0:1788204091.908584:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204091.908588:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204091.908592:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204091.908603:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187913472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204091.908613:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788204091.908613:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788204091.908619:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204091.908621:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204091.908626:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5546 00000100:00080000:1.0:1788204091.908631:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204091.908634:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204091.908639:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204091.908648:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187913600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204091.908659:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204091/real 1788204091] req@ffff8e828d720000 x1875062187913088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204091.908674:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187913088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204091.908681:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204091.908683:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204091.908686:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187913088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204091.908703:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187913216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204091.908718:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204091.908725:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204091/real 1788204091] req@ffff8e828d721500 x1875062187913216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204091.908732:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187913216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204091.908744:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204091.908747:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204091.908751:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187913216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204091.908756:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187913344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204091.908775:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204091.908786:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204091/real 1788204091] req@ffff8e828d723800 x1875062187913344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204091.908795:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187913344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204091.908800:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204091.908803:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204091.908805:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187913344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204091.908809:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187913472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204091.908822:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204091.908828:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204091/real 1788204091] req@ffff8e828d720e00 x1875062187913472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204091.908835:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187913472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204091.908839:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204091.908840:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204091.908843:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187913472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204091.908848:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187913600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204091.908886:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204091.908898:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204091/real 1788204091] req@ffff8e828d722680 x1875062187913600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204091.908908:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187913600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204091.908913:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204091.908915:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204091.908917:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187913600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204092.612472:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204093.636505:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204094.660523:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204095.684562:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204096.708475:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204097.028468:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204097.028480:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204097.028486:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204097.028498:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5552 00000100:00080000:2.0:1788204097.028512:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204097.028520:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204097.028565:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187913728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204097.028605:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204097.028609:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204097.028611:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204097.028616:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5552 00000100:00080000:2.0:1788204097.028619:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204097.028623:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204097.028635:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204097.028648:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187913856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204097.028662:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204097.028666:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204097.028668:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204097.028672:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5552 00000100:00080000:2.0:1788204097.028675:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204097.028680:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204097.028685:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204097.028700:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187913984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204097.028703:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187913728:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204097.028732:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204097.028735:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204097.028737:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204097.028749:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5552 00000100:00080000:2.0:1788204097.028754:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204097.028759:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204097.028766:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204097.028815:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187914112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204097.028816:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204097.028832:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204097.028836:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204097.028839:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204097.028844:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5552 00000100:00080000:2.0:1788204097.028848:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204097.028853:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204097.028858:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204097.028869:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187914240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204097.028880:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204097/real 1788204097] req@ffff8e82894e4000 x1875062187913728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204097.028902:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187913728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204097.028917:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204097.028921:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204097.028926:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187913728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204097.028934:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187913856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204097.028959:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204097.028972:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204097/real 1788204097] req@ffff8e82894e7800 x1875062187913856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204097.028983:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187913856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204097.028991:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204097.028994:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204097.028997:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187913856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204097.029021:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187913984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204097.029049:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204097.029059:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204097/real 1788204097] req@ffff8e82894e6300 x1875062187913984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204097.029070:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187913984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204097.029077:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204097.029080:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204097.029083:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187913984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204097.029089:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187914112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204097.029121:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204097.029131:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204097/real 1788204097] req@ffff8e82894e4a80 x1875062187914112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204097.029141:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187914112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204097.029147:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204097.029150:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204097.029153:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187914112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204097.029158:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187914240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204097.029174:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204097.029181:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204097/real 1788204097] req@ffff8e82894e4e00 x1875062187914240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204097.029192:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187914240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204097.029199:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204097.029202:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204097.029205:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187914240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204097.732678:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204098.756489:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204099.780543:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204100.804529:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204101.828494:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204102.148428:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204102.148442:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204102.148448:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204102.148461:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5557 00000100:00080000:3.0:1788204102.148475:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204102.148485:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204102.148532:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187914368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204102.148565:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204102.148570:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204102.148572:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204102.148577:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5557 00000100:00080000:3.0:1788204102.148583:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204102.148588:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204102.148600:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204102.148605:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187914368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204102.148617:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187914496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204102.148631:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204102.148635:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204102.148638:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204102.148643:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5557 00000100:00080000:3.0:1788204102.148647:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204102.148652:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204102.148658:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204102.148670:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187914624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204102.148696:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204102.148700:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204102.148703:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204102.148708:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5557 00000100:00080000:3.0:1788204102.148712:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204102.148718:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204102.148726:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204102.148737:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187914496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204102.148745:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187914752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204102.148759:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204102.148763:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204102.148766:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788204102.148772:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187914624:192.168.201.121@tcp:38:kworker.0 00000100:00080000:3.0:1788204102.148773:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5557 00000100:00080000:3.0:1788204102.148783:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204102.148797:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204102.148800:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187914752:192.168.201.121@tcp:8:kworker.0 00000008:00100000:3.0:1788204102.148814:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204102.148835:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187914880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204102.148881:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187914880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204102.852549:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204103.876484:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204104.900513:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788204105.220730:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788204105.220778:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788204105.220796:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204105.220841:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204105.220849:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204105.220855:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204105.220864:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:0.0:1788204105.221033:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204102/real 1788204105] req@ffff8e8289539500 x1875062187914880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204105.221059:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187914880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204105.221072:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204105.221077:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204105.221082:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187914880:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204105.221091:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204102/real 1788204105] req@ffff8e8289539f80 x1875062187914752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204105.221103:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187914752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204105.221110:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204105.221113:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204105.221117:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187914752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204105.221125:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204102/real 1788204105] req@ffff8e828953a680 x1875062187914496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204105.221135:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187914496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204105.221142:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204105.221145:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204105.221148:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187914496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204105.221154:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204102/real 1788204105] req@ffff8e828953b100 x1875062187914624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204105.221165:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187914624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204105.221171:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204105.221175:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204105.221177:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187914624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204105.221183:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204102/real 1788204105] req@ffff8e8289538a80 x1875062187914368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204105.221193:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187914368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204105.221200:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204105.221203:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204105.221206:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187914368:192.168.201.121@tcp:250:kworker.0 00010000:00010000:0.0:1788204105.924471:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204106.948484:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204107.268433:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204107.268448:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204107.268453:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204107.268465:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5562 00000100:00080000:3.0:1788204107.268479:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204107.268488:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204107.268533:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187915008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204107.268565:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204107.268569:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204107.268572:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204107.268577:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5562 00000100:00080000:3.0:1788204107.268581:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204107.268586:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204107.268609:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204107.268629:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187915136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204107.268642:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204107.268646:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204107.268650:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788204107.268651:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187915008:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204107.268657:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5562 00000100:00080000:3.0:1788204107.268664:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204107.268669:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204107.268675:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204107.268690:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187915264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204107.268699:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204107.268703:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204107.268706:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204107.268710:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5562 00000100:00080000:3.0:1788204107.268714:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204107.268718:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204107.268724:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204107.268738:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187915392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204107.268742:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788204107.268753:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204107.268757:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204107.268759:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204107.268765:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5562 00000100:00080000:3.0:1788204107.268770:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204107.268774:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204107.268779:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204107.268791:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187915520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204107.268802:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204107/real 1788204107] req@ffff8e8289539880 x1875062187915008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204107.268848:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187915008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204107.268861:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204107.268866:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204107.268870:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187915008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204107.268911:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187915136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204107.268941:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204107.268955:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204107/real 1788204107] req@ffff8e828953bb80 x1875062187915136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204107.268968:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187915136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204107.268977:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204107.268981:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204107.268985:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187915136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204107.268991:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187915264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204107.269021:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204107.269032:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204107/real 1788204107] req@ffff8e8289538380 x1875062187915264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204107.269044:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187915264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204107.269053:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204107.269056:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204107.269059:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187915264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204107.269066:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187915392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204107.269085:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204107.269095:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204107/real 1788204107] req@ffff8e8289538a80 x1875062187915392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204107.269107:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187915392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204107.269117:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204107.269121:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204107.269124:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187915392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204107.269130:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187915520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204107.269162:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204107.269173:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204107/real 1788204107] req@ffff8e828953b100 x1875062187915520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204107.269193:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187915520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204107.269201:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204107.269204:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204107.269208:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187915520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204107.972512:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204108.996454:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204110.020467:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204111.044436:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204112.068454:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204112.388398:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204112.388408:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204112.388412:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204112.388422:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5567 00000100:00080000:3.0:1788204112.388433:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204112.388441:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204112.388477:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187915648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204112.388515:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204112.388519:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204112.388521:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204112.388526:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5567 00000100:00080000:3.0:1788204112.388529:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204112.388534:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204112.388545:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204112.388560:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187915648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204112.388562:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187915776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204112.388576:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204112.388579:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204112.388581:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204112.388589:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5567 00000100:00080000:3.0:1788204112.388594:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204112.388599:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204112.388606:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204112.388618:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187915904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204112.388633:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788204112.388638:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788204112.388640:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204112.388643:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204112.388648:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5567 00000100:00080000:3.0:1788204112.388653:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204112.388657:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204112.388661:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204112.388670:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187916032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204112.388677:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204112.388680:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788204112.388683:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204112/real 1788204112] req@ffff8e828953a680 x1875062187915648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204112.388683:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204112.388688:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5567 00000100:00080000:3.0:1788204112.388693:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204112.388699:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204112.388705:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204112.388717:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187915648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204112.388718:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187916160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204112.388732:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204112.388738:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204112.388743:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187915648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204112.388765:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187915776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204112.388788:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204112.388800:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204112/real 1788204112] req@ffff8e8289539f80 x1875062187915776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204112.388820:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187915776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204112.388829:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204112.388832:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204112.388836:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187915776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204112.388843:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187915904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204112.388876:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204112.388889:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204112/real 1788204112] req@ffff8e8289539500 x1875062187915904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204112.388901:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187915904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204112.388910:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204112.388914:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204112.388917:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187915904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204112.388924:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187916032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204112.388945:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204112.388958:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204112/real 1788204112] req@ffff8e828953b100 x1875062187916032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204112.388970:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187916032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204112.388983:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204112.388986:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204112.388989:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187916032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204112.388995:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187916160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204112.389013:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204112.389025:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204112/real 1788204112] req@ffff8e8289538a80 x1875062187916160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204112.389036:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187916160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204112.389045:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204112.389049:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204112.389052:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187916160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204113.092518:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204114.116467:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204115.140498:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204116.164502:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204117.188531:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204117.508487:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204117.508500:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204117.508505:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204117.508515:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5572 00000100:00080000:1.0:1788204117.508541:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204117.508550:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204117.508589:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187916288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204117.508619:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204117.508622:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204117.508623:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204117.508626:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5572 00000100:00080000:1.0:1788204117.508629:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204117.508633:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204117.508640:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204117.508656:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187916416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204117.508664:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204117.508667:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204117.508668:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788204117.508673:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187916288:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204117.508674:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5572 00000100:00080000:1.0:1788204117.508679:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204117.508683:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204117.508686:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204117.508694:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187916544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204117.508700:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204117.508703:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204117.508704:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204117.508707:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5572 00000100:00080000:1.0:1788204117.508708:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204117.508711:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204117.508714:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204117.508720:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187916672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204117.508726:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204117.508729:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204117.508730:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204117.508732:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5572 00000100:00080000:1.0:1788204117.508735:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788204117.508740:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788204117.508741:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204117.508746:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204117.508753:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187916800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204117.508793:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204117/real 1788204117] req@ffff8e828d722300 x1875062187916288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204117.508814:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187916288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204117.508831:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204117.508834:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204117.508838:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187916288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204117.508857:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187916416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204117.508876:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204117.508886:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204117/real 1788204117] req@ffff8e828d722d80 x1875062187916416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204117.508894:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187916416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204117.508899:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204117.508901:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204117.508904:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187916416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204117.508907:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187916544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204117.508917:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204117.508923:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204117/real 1788204117] req@ffff8e828d721180 x1875062187916544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204117.508930:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187916544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204117.508935:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204117.508937:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204117.508939:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187916544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204117.508942:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187916672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204117.508952:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204117.508958:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204117/real 1788204117] req@ffff8e828d720380 x1875062187916672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204117.508965:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187916672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204117.508970:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204117.508972:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204117.508973:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187916672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204117.508976:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187916800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204117.508991:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204117.509001:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204117/real 1788204117] req@ffff8e828d720a80 x1875062187916800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204117.509013:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187916800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204117.509021:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204117.509025:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204117.509029:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187916800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204118.212482:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204119.236571:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204120.260535:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204121.284570:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204122.308472:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204122.628451:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204122.628461:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204122.628465:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204122.628477:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5577 00000100:00080000:3.0:1788204122.628488:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204122.628495:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204122.628531:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187916928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204122.628559:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204122.628564:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204122.628566:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204122.628570:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5577 00000100:00080000:3.0:1788204122.628574:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204122.628579:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204122.628590:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204122.628604:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187917056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204122.628616:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788204122.628617:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187916928:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204122.628624:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204122.628628:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204122.628633:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5577 00000100:00080000:3.0:1788204122.628636:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204122.628640:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204122.628645:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204122.628658:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187917184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204122.628668:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204122.628671:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204122.628673:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204122.628677:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5577 00000100:00080000:3.0:1788204122.628681:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204122.628685:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204122.628692:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788204122.628698:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204122.628707:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187917312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204122.628718:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204122.628722:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204122.628723:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204122.628727:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5577 00000100:00080000:3.0:1788204122.628730:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204122.628735:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204122.628741:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204122.628745:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204122/real 1788204122] req@ffff8e8289538380 x1875062187916928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204122.628756:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187917440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204122.628762:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187916928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204122.628779:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204122.628784:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204122.628789:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187916928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204122.628819:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187917056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204122.628844:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204122.628859:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204122/real 1788204122] req@ffff8e828953bb80 x1875062187917056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204122.628872:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187917056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204122.628881:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204122.628885:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204122.628888:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187917056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204122.628894:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187917184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204122.628915:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204122.628925:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204122/real 1788204122] req@ffff8e8289539880 x1875062187917184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204122.628933:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187917184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204122.628939:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204122.628941:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204122.628953:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187917184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204122.628957:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187917312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204122.628970:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204122.628977:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204122/real 1788204122] req@ffff8e8289538a80 x1875062187917312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204122.628985:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187917312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204122.628990:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204122.628992:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204122.628993:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187917312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204122.628997:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187917440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204122.629007:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204122.629012:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204122/real 1788204122] req@ffff8e828953b100 x1875062187917440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204122.629021:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187917440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204122.629036:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204122.629039:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204122.629043:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187917440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204123.332480:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204124.356474:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204125.380467:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204126.404487:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204127.428473:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204127.748430:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204127.748440:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204127.748444:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204127.748452:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5582 00000100:00080000:1.0:1788204127.748462:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204127.748468:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204127.748502:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187917568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204127.748526:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204127.748529:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204127.748530:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204127.748533:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5582 00000100:00080000:1.0:1788204127.748535:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204127.748538:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204127.748544:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204127.748553:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187917696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204127.748560:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204127.748563:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204127.748564:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204127.748566:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5582 00000100:00080000:1.0:1788204127.748568:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204127.748570:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204127.748574:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204127.748582:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187917824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204127.748589:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204127.748591:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788204127.748592:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187917568:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204127.748593:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204127.748601:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5582 00000100:00080000:1.0:1788204127.748604:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204127.748606:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204127.748609:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204127.748617:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187917952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204127.748624:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204127.748626:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204127.748627:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204127.748630:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5582 00000100:00080000:1.0:1788204127.748646:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204127.748649:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204127.748652:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788204127.748656:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204127.748663:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187918080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204127.748696:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204127/real 1788204127] req@ffff8e828d720a80 x1875062187917568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204127.748710:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187917568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204127.748719:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204127.748721:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204127.748725:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187917568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204127.748730:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187917696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204127.748744:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204127.748751:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204127/real 1788204127] req@ffff8e828d720380 x1875062187917696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204127.748758:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187917696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204127.748764:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204127.748766:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204127.748768:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187917696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204127.748787:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187917824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204127.748797:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204127.748803:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204127/real 1788204127] req@ffff8e828d721180 x1875062187917824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204127.748810:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187917824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204127.748816:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204127.748817:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204127.748819:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187917824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204127.748823:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187917952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204127.748833:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204127.748839:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204127/real 1788204127] req@ffff8e828d722d80 x1875062187917952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204127.748851:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187917952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204127.748858:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204127.748861:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204127.748865:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187917952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204127.748871:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187918080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204127.748891:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204127.748902:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204127/real 1788204127] req@ffff8e828d722300 x1875062187918080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204127.748911:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187918080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204127.748917:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204127.748918:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204127.748921:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187918080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204128.452453:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204129.476505:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204130.500539:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204131.524470:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204132.548560:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204132.868422:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204132.868440:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204132.868445:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204132.868456:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5587 00000100:00080000:3.0:1788204132.868479:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204132.868489:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204132.868541:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187918208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204132.868569:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204132.868572:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204132.868574:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204132.868577:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5587 00000100:00080000:3.0:1788204132.868582:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204132.868585:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204132.868595:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204132.868608:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187918336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204132.868616:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204132.868619:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204132.868620:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204132.868623:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5587 00000100:00080000:3.0:1788204132.868625:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204132.868628:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204132.868633:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204132.868640:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187918208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204132.868650:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187918464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204132.868660:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204132.868663:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204132.868665:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204132.868668:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5587 00000100:00080000:3.0:1788204132.868672:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204132.868675:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204132.868679:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204132.868691:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187918592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204132.868705:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204132.868708:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204132.868709:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204132.868712:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5587 00000100:00080000:3.0:1788204132.868714:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204132.868717:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204132.868720:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204132.868727:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187918720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204132.868746:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204132.868795:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204132/real 1788204132] req@ffff8e8289539500 x1875062187918208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204132.868813:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187918208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204132.868826:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204132.868831:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204132.868836:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187918208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204132.868844:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187918336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204132.868867:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204132.868881:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204132/real 1788204132] req@ffff8e8289539f80 x1875062187918336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204132.868893:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187918336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204132.868901:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204132.868905:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204132.868908:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187918336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204132.868932:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187918464:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204132.868952:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204132.868962:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204132/real 1788204132] req@ffff8e828953a680 x1875062187918464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204132.868974:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187918464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204132.868982:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204132.868985:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204132.868989:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187918464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204132.868995:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187918592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204132.869015:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204132.869028:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204132/real 1788204132] req@ffff8e828953b100 x1875062187918592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204132.869040:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187918592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204132.869048:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204132.869051:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204132.869055:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187918592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204132.869061:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187918720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204132.869082:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204132.869092:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204132/real 1788204132] req@ffff8e8289538a80 x1875062187918720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204132.869100:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187918720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204132.869105:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204132.869107:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204132.869109:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187918720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204133.572507:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204134.596491:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204135.620560:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204136.644508:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204137.668477:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204137.988443:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204137.988454:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204137.988459:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204137.988470:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5593 00000100:00080000:3.0:1788204137.988486:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204137.988495:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204137.988542:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187918848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204137.988577:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204137.988582:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204137.988584:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204137.988589:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5593 00000100:00080000:3.0:1788204137.988594:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204137.988598:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204137.988609:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204137.988624:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187918976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204137.988633:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204137.988636:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204137.988640:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788204137.988642:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187918848:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204137.988644:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5593 00000100:00080000:3.0:1788204137.988649:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204137.988655:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204137.988659:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204137.988667:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187919104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204137.988674:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204137.988677:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204137.988678:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204137.988681:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5593 00000100:00080000:3.0:1788204137.988683:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204137.988685:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204137.988688:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204137.988697:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187919232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204137.988707:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204137.988711:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204137.988714:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204137.988719:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5593 00000100:00080000:3.0:1788204137.988723:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204137.988727:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:0.0:1788204137.988733:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:3.0:1788204137.988735:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204137.988751:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187919360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204137.988805:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204137/real 1788204137] req@ffff8e8289539880 x1875062187918848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204137.988828:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187918848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204137.988841:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204137.988846:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204137.988850:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187918848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204137.988880:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187918976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204137.988903:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204137.988917:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204137/real 1788204137] req@ffff8e828953bb80 x1875062187918976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204137.988930:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187918976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204137.988939:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204137.988942:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204137.988946:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187918976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204137.988952:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187919104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204137.988972:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204137.988983:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204137/real 1788204137] req@ffff8e8289538380 x1875062187919104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204137.988994:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187919104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204137.989008:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204137.989011:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204137.989015:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187919104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204137.989021:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187919232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204137.989041:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204137.989051:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204137/real 1788204137] req@ffff8e8289538a80 x1875062187919232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204137.989079:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187919232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204137.989088:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204137.989091:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204137.989095:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187919232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204137.989101:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187919360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204137.989121:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204137.989133:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204137/real 1788204137] req@ffff8e828953b100 x1875062187919360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204137.989147:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187919360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204137.989155:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204137.989159:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204137.989163:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187919360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204138.692545:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204139.716529:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204140.740510:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204141.764473:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204142.788528:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204143.108417:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204143.108428:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204143.108434:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204143.108445:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5598 00000100:00080000:1.0:1788204143.108457:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204143.108465:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204143.108507:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187919488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204143.108531:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204143.108534:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204143.108536:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204143.108539:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5598 00000100:00080000:1.0:1788204143.108542:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204143.108545:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204143.108554:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204143.108564:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187919616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204143.108575:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204143.108579:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204143.108581:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204143.108585:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5598 00000100:00080000:1.0:1788204143.108590:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204143.108594:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204143.108601:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204143.108602:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187919488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204143.108619:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187919744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204143.108635:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204143.108637:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204143.108639:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204143.108642:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5598 00000100:00080000:1.0:1788204143.108646:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204143.108649:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204143.108654:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204143.108663:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187919872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204143.108670:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204143.108673:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204143.108674:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204143.108676:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5598 00000100:00080000:1.0:1788204143.108679:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204143.108681:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:0.0:1788204143.108685:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:1.0:1788204143.108686:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204143.108696:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187920000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204143.108737:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204143/real 1788204143] req@ffff8e828d722680 x1875062187919488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204143.108753:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187919488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204143.108763:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204143.108766:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204143.108769:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187919488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204143.108774:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187919616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204143.108790:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204143.108798:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204143/real 1788204143] req@ffff8e828d720e00 x1875062187919616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204143.108806:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187919616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204143.108812:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204143.108814:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204143.108817:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187919616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204143.108864:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187919744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204143.108886:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204143.108895:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204143/real 1788204143] req@ffff8e828d723800 x1875062187919744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204143.108904:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187919744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204143.108909:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204143.108911:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204143.108914:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187919744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204143.108917:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187919872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204143.108929:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204143.108934:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204143/real 1788204143] req@ffff8e828d721500 x1875062187919872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204143.108942:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187919872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204143.108947:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204143.108948:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204143.108950:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187919872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204143.108954:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187920000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204143.109004:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204143.109019:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204143/real 1788204143] req@ffff8e828d720000 x1875062187920000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204143.109033:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187920000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204143.109042:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204143.109045:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204143.109049:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187920000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204143.812435:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204144.836503:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204145.860489:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204146.884543:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204147.908527:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204148.228445:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204148.228459:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204148.228465:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204148.228480:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5603 00000100:00080000:3.0:1788204148.228497:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204148.228509:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204148.228555:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187920128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204148.228586:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204148.228592:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204148.228595:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204148.228602:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5603 00000100:00080000:3.0:1788204148.228608:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204148.228615:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204148.228626:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187920128:192.168.201.121@tcp:250:kworker.0 00000008:00100000:3.0:1788204148.228629:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204148.228659:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187920256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204148.228682:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204148.228686:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204148.228688:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204148.228693:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5603 00000100:00080000:3.0:1788204148.228699:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204148.228704:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204148.228710:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788204148.228712:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204148.228767:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204148/real 1788204148] req@ffff8e828953a680 x1875062187920128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204148.228788:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187920128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204148.228800:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204148.228806:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204148.228811:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187920384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204148.228815:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187920128:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204148.228830:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204148.228835:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204148.228837:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204148.228842:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5603 00000100:00080000:3.0:1788204148.228848:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788204148.228849:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187920256:192.168.201.121@tcp:38:kworker.0 00000100:00080000:3.0:1788204148.228858:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:2.0:1788204148.228908:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204148.228924:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204148/real 1788204148] req@ffff8e8289539f80 x1875062187920256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204148.228939:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187920256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204148.228949:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204148.228953:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204148.228957:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187920256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204148.228965:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187920384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204148.228985:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204148.228994:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204148/real 1788204148] req@ffff8e8289539500 x1875062187920384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204148.229008:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187920384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204148.229014:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204148.229017:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204148.229020:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187920384:192.168.201.121@tcp:38:kworker.0 00000008:00100000:3.0:1788204148.229035:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204148.229052:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187920512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204148.229112:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204148.229116:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204148.229118:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204148.229124:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5603 00000100:00080000:3.0:1788204148.229130:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204148.229135:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204148.229141:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204148.229155:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187920512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204148.229156:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187920640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204148.229196:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204148.229207:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204148/real 1788204148] req@ffff8e8289539500 x1875062187920512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204148.229215:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187920512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204148.229221:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204148.229223:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204148.229226:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187920512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204148.229235:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187920640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204148.229247:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204148.229253:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204148/real 1788204148] req@ffff8e8289539f80 x1875062187920640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204148.229261:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187920640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204148.229271:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204148.229273:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204148.229275:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187920640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204148.932541:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204149.956533:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204150.980483:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204152.004523:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204153.028470:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204153.348433:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204153.348446:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204153.348451:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204153.348463:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5608 00000100:00080000:1.0:1788204153.348476:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204153.348484:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204153.348533:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187920768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204153.348572:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204153.348576:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204153.348577:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204153.348582:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5608 00000100:00080000:1.0:1788204153.348586:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204153.348590:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204153.348601:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204153.348614:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187920896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204153.348627:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204153.348630:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204153.348632:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204153.348637:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5608 00000100:00080000:1.0:1788204153.348642:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788204153.348649:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187920768:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204153.348651:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204153.348666:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204153.348685:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187921024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204153.348698:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204153.348702:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204153.348705:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204153.348709:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5608 00000100:00080000:1.0:1788204153.348721:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788204153.348726:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788204153.348731:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204153.348739:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204153.348752:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187921152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204153.348762:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204153.348766:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788204153.348769:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204153/real 1788204153] req@ffff8e828d720000 x1875062187920768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204153.348770:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204153.348785:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5608 00000100:00080000:1.0:1788204153.348815:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788204153.348819:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187920768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204153.348832:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:0.0:1788204153.348837:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204153.348841:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000008:00100000:1.0:1788204153.348847:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204153.348848:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187920768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204153.348857:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187920896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204153.348876:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187921280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204153.348888:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204153.348912:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204153/real 1788204153] req@ffff8e828d721500 x1875062187920896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204153.348935:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187920896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204153.348943:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204153.348947:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204153.348949:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187920896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204153.348980:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187921024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204153.349001:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204153.349011:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204153/real 1788204153] req@ffff8e828d723800 x1875062187921024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204153.349029:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187921024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204153.349036:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204153.349039:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204153.349041:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187921024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204153.349047:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187921152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204153.349071:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204153.349082:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204153/real 1788204153] req@ffff8e828d720e00 x1875062187921152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204153.349092:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187921152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204153.349104:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204153.349107:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204153.349110:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187921152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204153.349115:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187921280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204153.349133:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204153.349149:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204153/real 1788204153] req@ffff8e828d722680 x1875062187921280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204153.349156:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187921280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204153.349161:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204153.349163:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204153.349165:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187921280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204154.052495:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204155.076471:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204156.100503:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204157.124515:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204158.148501:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204158.468473:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204158.468487:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204158.468492:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204158.468505:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5613 00000100:00080000:1.0:1788204158.468518:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204158.468527:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204158.468587:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187921408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204158.468622:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204158.468627:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204158.468629:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204158.468634:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5613 00000100:00080000:1.0:1788204158.468638:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204158.468642:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204158.468655:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204158.468670:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187921536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204158.468681:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187921408:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204158.468682:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204158.468690:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204158.468694:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204158.468698:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5613 00000100:00080000:1.0:1788204158.468715:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204158.468720:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204158.468726:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204158.468738:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187921664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204158.468747:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204158.468751:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204158.468752:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204158.468756:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5613 00000100:00080000:1.0:1788204158.468777:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204158.468783:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:0.0:1788204158.468790:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:1.0:1788204158.468797:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204158.468813:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187921792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204158.468822:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204158.468825:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204158.468827:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204158.468831:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5613 00000100:00080000:1.0:1788204158.468835:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204158.468839:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204158.468845:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204158.468847:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204158/real 1788204158] req@ffff8e828d722300 x1875062187921408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204158.468859:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187921920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204158.468869:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187921408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204158.468889:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204158.468894:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204158.468898:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187921408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204158.468927:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187921536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204158.468958:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204158.468970:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204158/real 1788204158] req@ffff8e828d722d80 x1875062187921536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204158.468982:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187921536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204158.468991:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204158.468994:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204158.468998:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187921536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204158.469003:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187921664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204158.469020:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204158.469031:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204158/real 1788204158] req@ffff8e828d721180 x1875062187921664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204158.469042:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187921664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204158.469054:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204158.469057:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204158.469060:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187921664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204158.469066:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187921792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204158.469085:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204158.469094:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204158/real 1788204158] req@ffff8e828d720380 x1875062187921792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204158.469105:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187921792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204158.469112:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204158.469115:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204158.469118:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187921792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204158.469124:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187921920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204158.469141:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204158.469151:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204158/real 1788204158] req@ffff8e828d720a80 x1875062187921920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204158.469161:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187921920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204158.469169:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204158.469172:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204158.469176:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187921920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204159.172519:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204160.196488:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204161.220492:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204162.244493:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204163.268467:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204163.588418:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204163.588433:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204163.588438:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204163.588453:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5618 00000100:00080000:1.0:1788204163.588468:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204163.588478:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204163.588530:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187922048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204163.588564:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204163.588571:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204163.588572:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204163.588577:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5618 00000100:00080000:1.0:1788204163.588581:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204163.588586:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204163.588598:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204163.588622:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187922048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204163.588628:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187922176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204163.588641:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204163.588645:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204163.588648:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204163.588652:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5618 00000100:00080000:1.0:1788204163.588656:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204163.588660:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204163.588665:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204163.588679:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187922304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204163.588688:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204163.588692:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204163.588693:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204163.588697:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5618 00000100:00080000:1.0:1788204163.588701:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788204163.588703:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788204163.588708:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204163.588715:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204163.588729:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187922432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204163.588739:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204163.588743:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204163.588745:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788204163.588749:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204163/real 1788204163] req@ffff8e828d720a80 x1875062187922048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204163.588750:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5618 00000100:00080000:1.0:1788204163.588758:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204163.588764:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204163.588769:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204163.588770:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187922048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204163.588785:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187922560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204163.588790:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204163.588797:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204163.588802:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187922048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204163.588827:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187922176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204163.588844:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204163.588851:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204163/real 1788204163] req@ffff8e828d720380 x1875062187922176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204163.588859:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187922176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204163.588865:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204163.588867:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204163.588870:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187922176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204163.588876:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187922304:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204163.588895:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204163.588919:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204163/real 1788204163] req@ffff8e828d721180 x1875062187922304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204163.588931:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187922304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204163.588939:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204163.588943:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204163.588946:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187922304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204163.588952:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187922432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204163.588971:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204163.588978:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204163/real 1788204163] req@ffff8e828d722d80 x1875062187922432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204163.588985:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187922432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204163.588991:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204163.588994:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204163.588997:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187922432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204163.589003:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187922560:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204163.589021:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204163.589032:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204163/real 1788204163] req@ffff8e828d722300 x1875062187922560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204163.589044:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187922560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204163.589052:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204163.589055:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204163.589059:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187922560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204164.292497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204165.316493:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204166.340478:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204167.364480:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204168.388556:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204168.708438:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204168.708447:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204168.708452:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204168.708462:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5623 00000100:00080000:1.0:1788204168.708472:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204168.708479:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204168.708520:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187922688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204168.708566:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204168.708570:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204168.708571:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204168.708576:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5623 00000100:00080000:1.0:1788204168.708579:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204168.708583:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204168.708591:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204168.708603:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187922816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204168.708609:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204168.708612:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204168.708613:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204168.708615:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5623 00000100:00080000:1.0:1788204168.708617:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204168.708619:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204168.708622:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204168.708624:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187922688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204168.708630:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187922944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204168.708638:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204168.708640:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204168.708641:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204168.708643:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5623 00000100:00080000:1.0:1788204168.708645:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204168.708647:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204168.708650:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204168.708656:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187923072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204168.708665:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204168.708668:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204168.708669:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204168.708671:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5623 00000100:00080000:1.0:1788204168.708673:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204168.708675:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204168.708678:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204168.708683:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187923200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204168.708757:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187922816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204168.708806:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187922944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204168.708827:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187923072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204168.708854:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187923200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204169.412468:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204170.436523:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204171.460510:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788204171.780542:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788204171.780605:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788204171.780627:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204171.780683:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204171.780695:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204171.780706:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204171.780714:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:0.0:1788204171.780935:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204168/real 1788204171] req@ffff8e828d723800 x1875062187922944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204171.780967:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187922944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204171.780981:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204171.780986:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204171.781007:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187922944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204171.781017:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204168/real 1788204171] req@ffff8e828d721500 x1875062187923072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204171.781030:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187923072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204171.781038:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204171.781042:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204171.781046:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187923072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204171.781053:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204168/real 1788204171] req@ffff8e828d720000 x1875062187923200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204171.781064:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187923200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204171.781072:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204171.781075:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204171.781078:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187923200:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204171.781085:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204168/real 1788204171] req@ffff8e828d722680 x1875062187922688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204171.781105:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187922688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204171.781114:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204171.781117:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204171.781120:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187922688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204171.781127:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204168/real 1788204171] req@ffff8e828d720e00 x1875062187922816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204171.781140:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187922816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204171.781149:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204171.781152:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204171.781155:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187922816:192.168.201.121@tcp:38:kworker.0 00010000:00010000:0.0:1788204172.484480:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204173.508555:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204173.828489:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204173.828497:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204173.828500:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204173.828508:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5628 00000100:00080000:3.0:1788204173.828517:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204173.828523:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204173.828565:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187923328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204173.828595:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204173.828599:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204173.828601:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204173.828605:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5628 00000100:00080000:3.0:1788204173.828609:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204173.828613:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204173.828623:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204173.828638:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187923456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204173.828646:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204173.828648:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204173.828650:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204173.828653:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5628 00000100:00080000:3.0:1788204173.828657:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204173.828660:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204173.828664:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204173.828688:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187923584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204173.828697:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204173.828701:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204173.828703:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204173.828707:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5628 00000100:00080000:3.0:1788204173.828710:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204173.828714:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204173.828719:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204173.828729:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187923712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204173.828742:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204173.828745:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204173.828746:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204173.828750:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5628 00000100:00080000:3.0:1788204173.828753:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204173.828756:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204173.828760:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204173.828770:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187923840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204173.829032:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187923328:192.168.201.121@tcp:250:kworker.0 00000800:00000100:0.0:1788204173.829095:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204173.829142:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204173/real 1788204173] req@ffff8e828953a680 x1875062187923328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204173.829171:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187923328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204173.829182:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204173.829185:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204173.829189:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187923328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204173.829199:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187923456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204173.829219:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204173.829230:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204173/real 1788204173] req@ffff8e828953b100 x1875062187923456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204173.829241:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187923456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204173.829248:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204173.829251:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204173.829254:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187923456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204173.829259:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187923584:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204173.829277:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204173.829287:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204173/real 1788204173] req@ffff8e8289538a80 x1875062187923584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204173.829298:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187923584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204173.829304:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204173.829353:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204173.829356:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187923584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204173.829362:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187923712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204173.829382:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204173.829393:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204173/real 1788204173] req@ffff8e8289538380 x1875062187923712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204173.829405:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187923712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204173.829413:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204173.829416:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204173.829419:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187923712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204173.829424:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187923840:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204173.829459:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204173.829469:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204173/real 1788204173] req@ffff8e828953bb80 x1875062187923840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204173.829480:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187923840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204173.829488:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204173.829491:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204173.829494:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187923840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204174.532494:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204175.556471:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204176.580469:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204177.604475:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204178.628453:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204178.948452:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204178.948466:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204178.948471:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204178.948485:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5634 00000100:00080000:0.0:1788204178.948499:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204178.948508:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204178.948559:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187923968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204178.948592:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204178.948596:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204178.948597:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204178.948602:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5634 00000100:00080000:0.0:1788204178.948606:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204178.948610:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204178.948622:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204178.948635:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187924096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204178.948645:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187923968:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204178.948647:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204178.948655:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204178.948657:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204178.948671:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5634 00000100:00080000:0.0:1788204178.948675:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204178.948679:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204178.948684:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204178.948696:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187924224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204178.948711:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204178.948715:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204178.948717:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204178.948721:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5634 00000100:00080000:0.0:1788204178.948725:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204178.948728:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:2.0:1788204178.948733:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788204178.948735:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204178.948748:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187924352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204178.948757:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204178.948760:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204178.948762:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204178.948766:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5634 00000100:00080000:0.0:1788204178.948769:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204178.948773:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204178.948778:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204178.948789:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204178/real 1788204178] req@ffff8e828793e300 x1875062187923968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204178.948799:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187924480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204178.948811:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187923968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204178.948823:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204178.948827:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204178.948831:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187923968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204178.948860:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187924096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204178.948881:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204178.948892:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204178/real 1788204178] req@ffff8e828793d500 x1875062187924096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204178.948903:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187924096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204178.948911:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204178.948914:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204178.948917:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187924096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204178.948923:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187924224:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204178.948940:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204178.948949:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204178/real 1788204178] req@ffff8e828793fb80 x1875062187924224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204178.948961:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187924224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204178.948969:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204178.948971:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204178.948973:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187924224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204178.948977:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187924352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204178.948993:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204178.948999:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204178/real 1788204178] req@ffff8e828793d880 x1875062187924352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204178.949006:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187924352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204178.949011:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204178.949012:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204178.949014:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187924352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204178.949017:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187924480:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204178.949026:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204178.949030:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204178/real 1788204178] req@ffff8e828793c380 x1875062187924480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204178.949037:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187924480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204178.949042:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204178.949043:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204178.949045:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187924480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204179.652478:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204180.676548:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204181.700501:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204182.724479:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204183.748506:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204184.068462:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204184.068477:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204184.068483:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204184.068499:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5639 00000100:00080000:0.0:1788204184.068514:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204184.068525:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204184.068572:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187924608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204184.068610:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204184.068614:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204184.068617:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204184.068625:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5639 00000100:00080000:0.0:1788204184.068631:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204184.068637:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204184.068650:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204184.068653:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187924608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204184.068669:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187924736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204184.068682:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204184.068685:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204184.068688:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204184.068692:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5639 00000100:00080000:0.0:1788204184.068696:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204184.068701:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204184.068706:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204184.068721:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187924864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204184.068730:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204184.068739:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204184.068744:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204184.068746:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204184.068752:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5639 00000100:00080000:0.0:1788204184.068756:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204184.068761:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204184.068768:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204184.068776:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204184/real 1788204184] req@ffff8e828793f480 x1875062187924608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204184.068786:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187924992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204184.068795:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187924608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204184.068804:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204184.068809:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204184.068809:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204184.068815:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204184.068816:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:0.0:1788204184.068826:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5639 00000100:00100000:2.0:1788204184.068826:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187924608:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204184.068834:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204184.068842:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204184.068848:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204184.068862:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187925120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204184.068863:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187924736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204184.068889:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204184.068908:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204184/real 1788204184] req@ffff8e828793dc00 x1875062187924736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204184.068918:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187924736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204184.068928:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204184.068931:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204184.068933:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187924736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204184.068937:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187924864:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204184.068950:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204184.068964:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204184/real 1788204184] req@ffff8e828793c000 x1875062187924864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204184.068972:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187924864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204184.068978:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204184.068981:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204184.068984:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187924864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204184.068991:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187924992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204184.069013:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204184.069026:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204184/real 1788204184] req@ffff8e828793ca80 x1875062187924992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204184.069039:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187924992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204184.069048:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204184.069051:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204184.069056:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187924992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204184.069071:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187925120:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204184.069088:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204184.069096:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204184/real 1788204184] req@ffff8e828793c700 x1875062187925120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204184.069104:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187925120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204184.069110:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204184.069112:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204184.069114:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187925120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204184.772499:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204185.796487:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204186.820519:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204187.844534:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204188.868527:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204189.188411:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204189.188425:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204189.188431:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204189.188444:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5644 00000100:00080000:0.0:1788204189.188457:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204189.188465:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204189.188512:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187925248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204189.188545:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204189.188549:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204189.188551:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204189.188556:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5644 00000100:00080000:0.0:1788204189.188560:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204189.188565:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204189.188577:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204189.188592:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187925376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204189.188599:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187925248:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204189.188606:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204189.188609:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204189.188611:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204189.188619:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5644 00000100:00080000:0.0:1788204189.188623:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204189.188627:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204189.188632:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204189.188645:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187925504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204189.188662:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204189.188666:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204189.188668:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204189.188672:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5644 00000100:00080000:0.0:1788204189.188676:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204189.188680:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204189.188686:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204189.188700:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187925632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204189.188700:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204189.188712:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204189.188716:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204189.188717:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204189.188721:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5644 00000100:00080000:0.0:1788204189.188734:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204189.188739:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204189.188744:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204189.188758:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204189/real 1788204189] req@ffff8e828793ce00 x1875062187925248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204189.188759:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187925760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204189.188802:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187925248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204189.188815:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204189.188819:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204189.188823:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187925248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204189.188853:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187925376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204189.188877:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204189.188891:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204189/real 1788204189] req@ffff8e828793f800 x1875062187925376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204189.188903:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187925376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204189.188911:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204189.188914:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204189.188918:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187925376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204189.188924:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187925504:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204189.188945:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204189.188956:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204189/real 1788204189] req@ffff8e828793e680 x1875062187925504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204189.188967:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187925504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204189.188974:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204189.188977:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204189.188980:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187925504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204189.188986:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187925632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204189.189006:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204189.189018:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204189/real 1788204189] req@ffff8e828793f100 x1875062187925632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204189.189029:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187925632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204189.189037:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204189.189040:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204189.189043:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187925632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204189.189049:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187925760:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204189.189067:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204189.189077:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204189/real 1788204189] req@ffff8e828793c700 x1875062187925760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204189.189088:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187925760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204189.189095:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204189.189098:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204189.189101:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187925760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204189.892509:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204190.916560:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204191.940461:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204192.964490:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204193.988450:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204194.308440:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204194.308447:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204194.308451:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204194.308460:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5649 00000100:00080000:1.0:1788204194.308470:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204194.308476:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204194.308514:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187925888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204194.308542:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204194.308545:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204194.308548:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204194.308552:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5649 00000100:00080000:1.0:1788204194.308556:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204194.308560:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204194.308569:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204194.308583:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187926016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204194.308589:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187925888:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204194.308600:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204194.308603:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204194.308604:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204194.308608:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5649 00000100:00080000:1.0:1788204194.308610:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204194.308614:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204194.308617:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204194.308627:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187926144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204194.308633:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204194.308635:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204194.308636:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204194.308639:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5649 00000100:00080000:1.0:1788204194.308643:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204194.308645:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204194.308648:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788204194.308654:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204194.308656:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187926272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204194.308663:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204194.308665:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204194.308667:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204194.308669:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5649 00000100:00080000:1.0:1788204194.308671:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204194.308674:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204194.308676:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204194.308682:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187926400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204194.308701:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204194/real 1788204194] req@ffff8e828d720e00 x1875062187925888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204194.308719:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187925888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204194.308729:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204194.308733:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204194.308736:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187925888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204194.308759:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187926016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204194.308793:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204194.308806:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204194/real 1788204194] req@ffff8e828d722680 x1875062187926016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204194.308817:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187926016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204194.308825:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204194.308828:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204194.308831:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187926016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204194.308836:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187926144:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204194.308854:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204194.308865:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204194/real 1788204194] req@ffff8e828d720000 x1875062187926144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204194.308875:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187926144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204194.308882:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204194.308885:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204194.308889:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187926144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204194.308894:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187926272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204194.308912:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204194.308923:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204194/real 1788204194] req@ffff8e828d721500 x1875062187926272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204194.308934:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187926272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204194.308941:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204194.308944:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204194.308947:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187926272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204194.308952:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187926400:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204194.308971:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204194.308981:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204194/real 1788204194] req@ffff8e828d723800 x1875062187926400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204194.308991:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187926400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204194.308999:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204194.309002:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204194.309005:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187926400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204195.012544:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204196.036583:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204197.060526:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204198.084545:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204199.108500:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204199.428407:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204199.428415:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204199.428417:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204199.428425:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5654 00000100:00080000:3.0:1788204199.428440:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204199.428445:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204199.428473:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187926528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204199.428492:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204199.428494:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204199.428495:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204199.428498:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5654 00000100:00080000:3.0:1788204199.428500:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204199.428502:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204199.428510:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204199.428523:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187926656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204199.428533:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187926528:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204199.428536:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204199.428541:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204199.428543:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204199.428547:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5654 00000100:00080000:3.0:1788204199.428551:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204199.428555:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204199.428561:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204199.428572:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187926784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204199.428578:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204199.428581:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204199.428582:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204199.428585:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5654 00000100:00080000:3.0:1788204199.428587:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788204199.428591:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788204199.428593:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204199.428597:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204199.428603:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187926912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204199.428609:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204199.428612:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204199.428613:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204199.428615:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5654 00000100:00080000:3.0:1788204199.428617:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204199.428620:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204199.428622:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204199.428628:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187927040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204199.428646:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204199/real 1788204199] req@ffff8e8289539880 x1875062187926528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204199.428662:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187926528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204199.428674:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204199.428676:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204199.428679:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187926528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204199.428697:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187926656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204199.428712:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204199.428720:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204199/real 1788204199] req@ffff8e828953bb80 x1875062187926656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204199.428727:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187926656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204199.428740:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204199.428742:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204199.428744:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187926656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204199.428748:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187926784:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204199.428759:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204199.428765:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204199/real 1788204199] req@ffff8e8289538380 x1875062187926784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204199.428771:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187926784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204199.428776:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204199.428777:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204199.428779:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187926784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204199.428783:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187926912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204199.428792:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204199.428797:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204199/real 1788204199] req@ffff8e8289538a80 x1875062187926912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204199.428803:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187926912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204199.428808:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204199.428809:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204199.428811:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187926912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204199.428814:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187927040:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204199.428826:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204199.428835:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204199/real 1788204199] req@ffff8e828953b100 x1875062187927040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204199.428853:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187927040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204199.428860:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204199.428863:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204199.428866:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187927040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204200.132500:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204201.156542:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204202.180478:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204203.204486:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204204.228496:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204204.548451:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204204.548460:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204204.548465:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204204.548480:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5659 00000100:00080000:1.0:1788204204.548490:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204204.548497:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204204.548536:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187927168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204204.548555:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204204.548557:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204204.548559:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204204.548562:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5659 00000100:00080000:1.0:1788204204.548565:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204204.548568:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204204.548575:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204204.548584:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187927296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204204.548590:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204204.548593:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204204.548595:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788204204.548596:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187927168:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204204.548598:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5659 00000100:00080000:1.0:1788204204.548603:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204204.548609:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204204.548613:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204204.548624:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187927424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204204.548634:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204204.548637:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204204.548639:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204204.548643:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5659 00000100:00080000:1.0:1788204204.548647:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204204.548651:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204204.548654:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788204204.548662:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204204.548664:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187927552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204204.548675:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204204.548678:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204204.548680:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204204.548684:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5659 00000100:00080000:1.0:1788204204.548687:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204204.548691:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204204.548695:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204204.548707:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187927680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204204.548711:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204204/real 1788204204] req@ffff8e828d722300 x1875062187927168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204204.548739:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187927168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204204.548749:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204204.548753:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204204.548757:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187927168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204204.548778:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187927296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204204.548798:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204204.548809:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204204/real 1788204204] req@ffff8e828d722d80 x1875062187927296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204204.548820:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187927296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204204.548828:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204204.548831:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204204.548834:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187927296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204204.548839:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187927424:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204204.548854:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204204.548862:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204204/real 1788204204] req@ffff8e828d721180 x1875062187927424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204204.548873:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187927424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204204.548886:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204204.548889:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204204.548892:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187927424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204204.548896:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187927552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204204.548946:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204204.548956:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204204/real 1788204204] req@ffff8e828d720380 x1875062187927552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204204.548967:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187927552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204204.548974:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204204.548977:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204204.548980:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187927552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204204.548985:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187927680:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204204.549000:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204204.549008:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204204/real 1788204204] req@ffff8e828d720a80 x1875062187927680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204204.549026:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187927680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204204.549037:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204204.549040:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204204.549044:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187927680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204205.252537:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204206.276542:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204207.300557:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204208.324601:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204209.348454:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204209.668470:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204209.668481:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204209.668485:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204209.668495:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5664 00000100:00080000:0.0:1788204209.668506:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204209.668513:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204209.668555:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187927808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204209.668582:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204209.668585:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204209.668586:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204209.668589:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5664 00000100:00080000:0.0:1788204209.668593:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204209.668597:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204209.668608:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204209.668621:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187927936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204209.668637:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204209.668639:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204209.668641:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204209.668643:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5664 00000100:00080000:0.0:1788204209.668646:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204209.668649:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204209.668652:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204209.668656:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187927808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204209.668664:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187928064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204209.668675:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204209.668678:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204209.668680:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204209.668683:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5664 00000100:00080000:0.0:1788204209.668687:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204209.668691:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204209.668697:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204209.668710:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187928192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204209.668718:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204209.668722:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204209.668724:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204209.668728:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5664 00000100:00080000:0.0:1788204209.668732:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788204209.668735:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204209.668740:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204209.668746:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204209.668760:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187928320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204209.668784:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204209/real 1788204209] req@ffff8e828793ca80 x1875062187927808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204209.668800:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187927808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204209.668808:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204209.668811:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204209.668828:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187927808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204209.668834:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187927936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204209.668853:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204209.668872:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204209/real 1788204209] req@ffff8e828793c000 x1875062187927936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204209.668880:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187927936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204209.668886:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204209.668888:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204209.668890:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187927936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204209.668910:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187928064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204209.668922:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204209.668929:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204209/real 1788204209] req@ffff8e828793dc00 x1875062187928064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204209.668936:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187928064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204209.668941:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204209.668943:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204209.668945:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187928064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204209.668948:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187928192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204209.668957:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204209.668962:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204209/real 1788204209] req@ffff8e828793f480 x1875062187928192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204209.668969:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187928192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204209.668973:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204209.668975:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204209.668976:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187928192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204209.668979:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187928320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204209.668988:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204209.668993:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204209/real 1788204209] req@ffff8e828793c380 x1875062187928320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204209.668999:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187928320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204209.669004:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204209.669006:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204209.669008:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187928320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204210.372547:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204211.396461:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204212.420545:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204213.444527:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204214.468506:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204214.788414:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204214.788428:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204214.788432:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204214.788444:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5669 00000100:00080000:0.0:1788204214.788456:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204214.788464:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204214.788512:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187928448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204214.788542:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204214.788545:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204214.788546:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204214.788549:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5669 00000100:00080000:0.0:1788204214.788553:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204214.788556:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204214.788566:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204214.788572:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187928448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204214.788581:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187928576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204214.788594:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204214.788597:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204214.788599:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204214.788603:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5669 00000100:00080000:0.0:1788204214.788607:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204214.788611:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204214.788616:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204214.788625:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187928704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204214.788632:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204214.788634:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204214.788635:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204214.788638:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5669 00000800:00000100:2.0:1788204214.788642:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204214.788643:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204214.788648:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204214.788652:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204214.788661:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187928832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204214.788667:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204214.788669:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204214.788670:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204214.788672:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5669 00000100:00080000:0.0:1788204214.788675:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204214.788677:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204214.788680:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204214.788687:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204214/real 1788204214] req@ffff8e828793d880 x1875062187928448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204214.788689:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187928960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204214.788720:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187928448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204214.788740:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204214.788743:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204214.788746:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187928448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204214.788766:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187928576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204214.788782:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204214.788790:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204214/real 1788204214] req@ffff8e828793fb80 x1875062187928576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204214.788797:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187928576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204214.788802:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204214.788804:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204214.788806:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187928576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204214.788816:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187928704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204214.788830:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204214.788836:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204214/real 1788204214] req@ffff8e828793d500 x1875062187928704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204214.788844:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187928704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204214.788848:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204214.788849:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204214.788851:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187928704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204214.788854:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187928832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204214.788864:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204214.788873:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204214/real 1788204214] req@ffff8e828793e300 x1875062187928832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204214.788883:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187928832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204214.788891:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204214.788894:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204214.788898:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187928832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204214.788903:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187928960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204214.788921:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204214.788932:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204214/real 1788204214] req@ffff8e828793c380 x1875062187928960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204214.788943:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187928960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204214.788951:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204214.788954:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204214.788957:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187928960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204215.492537:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204216.516543:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204217.540557:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204218.564534:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204219.588521:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204219.908476:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204219.908484:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204219.908487:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204219.908495:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5674 00000100:00080000:2.0:1788204219.908504:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204219.908509:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204219.908540:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187929088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204219.908564:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204219.908567:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204219.908569:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204219.908576:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5674 00000100:00080000:2.0:1788204219.908581:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204219.908585:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204219.908592:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204219.908603:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187929216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204219.908616:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187929088:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204219.908618:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204219.908630:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204219.908632:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204219.908649:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5674 00000100:00080000:2.0:1788204219.908654:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204219.908657:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204219.908661:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204219.908671:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187929344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204219.908678:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204219.908681:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204219.908683:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204219.908686:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5674 00000100:00080000:2.0:1788204219.908691:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204219.908696:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204219.908699:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204219.908710:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187929472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204219.908720:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204219.908722:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204219.908724:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788204219.908731:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204219.908735:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5674 00000100:00080000:2.0:1788204219.908740:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204219.908743:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204219.908746:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204219.908755:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187929600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204219.908797:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204219/real 1788204219] req@ffff8e82894e6d80 x1875062187929088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204219.908828:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187929088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204219.908839:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204219.908844:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204219.908848:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187929088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204219.908877:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187929216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204219.908902:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204219.908915:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204219/real 1788204219] req@ffff8e82894e5f80 x1875062187929216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204219.908927:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187929216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204219.908940:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204219.908943:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204219.908945:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187929216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204219.908951:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187929344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204219.908969:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204219.908980:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204219/real 1788204219] req@ffff8e82894e4380 x1875062187929344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204219.908991:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187929344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204219.908998:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204219.909001:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204219.909005:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187929344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204219.909011:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187929472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204219.909030:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204219.909041:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204219/real 1788204219] req@ffff8e82894e5c00 x1875062187929472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204219.909054:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187929472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204219.909060:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204219.909063:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204219.909066:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187929472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204219.909073:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187929600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204219.909094:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204219.909105:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204219/real 1788204219] req@ffff8e82894e7100 x1875062187929600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204219.909115:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187929600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204219.909123:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204219.909125:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204219.909128:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187929600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204220.612531:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204221.636488:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204222.660494:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204223.684446:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204224.708502:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204225.028494:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204225.028507:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204225.028513:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204225.028527:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5680 00000100:00080000:2.0:1788204225.028540:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204225.028559:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204225.028650:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187929728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204225.028672:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204225.028675:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204225.028677:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204225.028680:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5680 00000100:00080000:2.0:1788204225.028683:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204225.028686:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204225.028695:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204225.028708:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187929856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204225.028718:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204225.028726:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788204225.028729:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187929728:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204225.028729:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204225.028738:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5680 00000100:00080000:2.0:1788204225.028742:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204225.028747:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204225.028751:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204225.028761:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187929984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204225.028767:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204225.028770:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204225.028771:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204225.028774:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5680 00000100:00080000:2.0:1788204225.028776:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204225.028778:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204225.028781:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204225.028788:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187930112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204225.028793:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204225.028796:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204225.028797:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204225.028799:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5680 00000100:00080000:2.0:1788204225.028801:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204225.028803:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204225.028807:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788204225.028811:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204225.028816:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187930240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204225.028860:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204225/real 1788204225] req@ffff8e82894e7100 x1875062187929728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204225.028879:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187929728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204225.028890:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204225.028894:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204225.028898:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187929728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204225.028924:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187929856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204225.028942:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204225.028954:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204225/real 1788204225] req@ffff8e82894e5c00 x1875062187929856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204225.028964:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187929856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204225.028971:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204225.028975:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204225.028978:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187929856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204225.028983:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187929984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204225.029002:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204225.029012:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204225/real 1788204225] req@ffff8e82894e4380 x1875062187929984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204225.029023:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187929984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204225.029029:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204225.029032:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204225.029035:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187929984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204225.029039:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187930112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204225.029057:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204225.029066:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204225/real 1788204225] req@ffff8e82894e5f80 x1875062187930112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204225.029076:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187930112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204225.029082:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204225.029084:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204225.029088:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187930112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204225.029092:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187930240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204225.029108:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204225.029117:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204225/real 1788204225] req@ffff8e82894e6d80 x1875062187930240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204225.029127:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187930240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204225.029133:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204225.029136:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204225.029139:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187930240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204225.732522:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204226.756501:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204227.780540:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204228.804501:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204229.828481:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204230.148448:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204230.148481:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204230.148486:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204230.148496:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5685 00000100:00080000:3.0:1788204230.148507:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204230.148513:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204230.148553:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187930368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204230.148580:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204230.148583:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204230.148584:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204230.148588:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5685 00000100:00080000:3.0:1788204230.148590:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204230.148592:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204230.148600:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204230.148611:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187930496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204230.148622:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204230.148624:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204230.148626:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204230.148628:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5685 00000100:00080000:3.0:1788204230.148629:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204230.148631:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204230.148634:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204230.148641:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187930368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204230.148642:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187930624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204230.148653:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204230.148655:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204230.148656:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204230.148660:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5685 00000100:00080000:3.0:1788204230.148662:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204230.148666:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204230.148671:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204230.148692:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187930752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204230.148702:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204230.148706:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204230.148708:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788204230.148711:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788204230.148715:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5685 00000100:00080000:3.0:1788204230.148721:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204230.148725:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204230.148731:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204230.148743:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187930880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204230.148769:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204230/real 1788204230] req@ffff8e828953a680 x1875062187930368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204230.148797:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187930368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204230.148808:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204230.148812:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204230.148817:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187930368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204230.148825:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187930496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204230.148846:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204230.148856:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204230/real 1788204230] req@ffff8e8289539f80 x1875062187930496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204230.148863:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187930496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204230.148869:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204230.148871:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204230.148873:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187930496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204230.148894:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187930624:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204230.148905:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204230.148911:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204230/real 1788204230] req@ffff8e8289539500 x1875062187930624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204230.148918:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187930624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204230.148927:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204230.148929:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204230.148931:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187930624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204230.148934:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187930752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204230.148945:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204230.148954:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204230/real 1788204230] req@ffff8e828953b100 x1875062187930752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204230.148964:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187930752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204230.148971:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204230.148975:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204230.148978:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187930752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204230.148983:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187930880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204230.148999:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204230.149006:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204230/real 1788204230] req@ffff8e8289538a80 x1875062187930880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204230.149012:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187930880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204230.149020:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204230.149023:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204230.149026:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187930880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204230.852511:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204231.876522:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204232.900482:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204233.924527:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204234.948526:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204235.268441:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204235.268452:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204235.268458:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204235.268472:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5690 00000100:00080000:3.0:1788204235.268485:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204235.268494:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204235.268535:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187931008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204235.268564:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204235.268568:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204235.268570:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204235.268574:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5690 00000100:00080000:3.0:1788204235.268577:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204235.268581:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204235.268592:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204235.268606:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187931136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204235.268619:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187931008:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204235.268623:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204235.268628:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204235.268629:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204235.268632:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5690 00000100:00080000:3.0:1788204235.268639:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204235.268642:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204235.268648:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204235.268657:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187931264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204235.268666:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204235.268668:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204235.268670:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204235.268673:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5690 00000100:00080000:3.0:1788204235.268676:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204235.268679:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204235.268683:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204235.268691:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187931392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204235.268698:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204235.268701:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204235.268703:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204235.268706:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5690 00000100:00080000:3.0:1788204235.268709:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204235.268712:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204235.268716:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204235.268725:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187931520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204235.268755:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187931136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204235.268792:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187931264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204235.268809:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187931392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204235.268821:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187931520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204235.972519:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204236.996503:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204238.020600:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788204238.340617:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:1.0:1788204238.340659:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788204238.340677:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788204238.340721:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788204238.340740:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788204238.340759:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788204238.340773:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:0.0:1788204238.340785:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204235/real 1788204238] req@ffff8e828953bb80 x1875062187931136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204238.340826:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187931136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204238.340841:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204238.340847:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204238.340852:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187931136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204238.340862:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204235/real 1788204238] req@ffff8e8289539880 x1875062187931264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204238.340882:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187931264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204238.340891:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204238.340895:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204238.340898:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187931264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204238.340907:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204235/real 1788204238] req@ffff8e8289538a80 x1875062187931392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204238.340919:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187931392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204238.340927:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204238.340931:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204238.340935:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187931392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204238.340942:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204235/real 1788204238] req@ffff8e828953b100 x1875062187931520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204238.340953:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187931520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204238.340961:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204238.340965:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204238.340968:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187931520:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204238.340976:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204235/real 1788204238] req@ffff8e8289538380 x1875062187931008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204238.340988:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187931008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204238.340997:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204238.341000:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204238.341004:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187931008:192.168.201.121@tcp:250:kworker.0 00010000:00010000:0.0:1788204239.044519:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204240.068518:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204240.388502:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204240.388512:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204240.388517:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204240.388529:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5695 00000100:00080000:1.0:1788204240.388539:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204240.388549:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204240.388584:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187931648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204240.388607:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204240.388611:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204240.388613:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204240.388617:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5695 00000100:00080000:1.0:1788204240.388621:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204240.388626:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204240.388636:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204240.388664:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187931776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204240.388675:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204240.388679:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204240.388682:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204240.388688:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5695 00000100:00080000:1.0:1788204240.388693:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204240.388699:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204240.388708:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187931648:192.168.201.121@tcp:250:kworker.0 00000008:00100000:1.0:1788204240.388710:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204240.388727:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187931904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204240.388759:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204240.388763:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204240.388765:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204240.388770:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5695 00000100:00080000:1.0:1788204240.388775:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204240.388780:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204240.388787:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204240.388802:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187932032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204240.388812:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204240.388816:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204240.388818:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204240.388822:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5695 00000100:00080000:1.0:1788204240.388828:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788204240.388835:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788204240.388837:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204240.388846:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204240.388861:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187932160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204240.388928:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204240/real 1788204240] req@ffff8e828d720a80 x1875062187931648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204240.388955:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187931648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204240.388969:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204240.388975:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204240.388980:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187931648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204240.388991:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187931776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204240.389018:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204240.389031:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204240/real 1788204240] req@ffff8e828d720380 x1875062187931776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204240.389043:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187931776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204240.389052:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204240.389056:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204240.389059:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187931776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204240.389093:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187931904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204240.389114:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204240.389126:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204240/real 1788204240] req@ffff8e828d721180 x1875062187931904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204240.389138:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187931904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204240.389146:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204240.389149:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204240.389152:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187931904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204240.389159:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187932032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204240.389179:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204240.389191:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204240/real 1788204240] req@ffff8e828d722d80 x1875062187932032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204240.389202:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187932032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204240.389209:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204240.389213:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204240.389216:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187932032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204240.389222:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187932160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204240.389244:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204240.389255:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204240/real 1788204240] req@ffff8e828d722300 x1875062187932160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204240.389266:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187932160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204240.389274:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204240.389278:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204240.389293:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187932160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204241.092463:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204242.116494:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204243.141446:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204244.164486:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204245.188434:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204245.508391:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204245.508399:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204245.508402:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204245.508410:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5700 00000100:00080000:2.0:1788204245.508427:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204245.508434:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204245.508468:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187932288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204245.508488:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204245.508490:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204245.508492:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204245.508495:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5700 00000100:00080000:2.0:1788204245.508497:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204245.508499:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204245.508507:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204245.508525:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187932288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204245.508529:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187932416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204245.508541:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204245.508545:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204245.508547:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204245.508551:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5700 00000100:00080000:2.0:1788204245.508555:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204245.508558:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204245.508569:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204245.508580:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187932544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204245.508587:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204245.508590:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204245.508593:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204245.508595:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204245.508598:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5700 00000100:00080000:2.0:1788204245.508601:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204245.508603:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204245.508607:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204245.508614:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187932672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204245.508620:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204245.508623:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204245.508624:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204245.508627:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5700 00000100:00100000:0.0:1788204245.508628:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204245/real 1788204245] req@ffff8e82894e4e00 x1875062187932288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204245.508629:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204245.508635:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204245.508641:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204245.508650:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187932288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204245.508652:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187932800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204245.508671:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204245.508679:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204245.508683:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187932288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204245.508701:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187932416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204245.508717:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204245.508728:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204245/real 1788204245] req@ffff8e82894e4a80 x1875062187932416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204245.508740:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187932416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204245.508748:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204245.508751:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204245.508754:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187932416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204245.508760:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187932544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204245.508775:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204245.508785:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204245/real 1788204245] req@ffff8e82894e6300 x1875062187932544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204245.508795:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187932544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204245.508803:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204245.508806:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204245.508809:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187932544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204245.508815:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187932672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204245.508831:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204245.508839:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204245/real 1788204245] req@ffff8e82894e7800 x1875062187932672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204245.508845:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187932672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204245.508850:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204245.508852:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204245.508854:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187932672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204245.508858:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187932800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204245.508873:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204245.508882:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204245/real 1788204245] req@ffff8e82894e4000 x1875062187932800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204245.508892:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187932800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204245.508900:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204245.508903:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204245.508906:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187932800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204246.214406:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204247.236539:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204248.260478:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204249.284493:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204250.308476:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204250.628472:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204250.628485:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204250.628490:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204250.628502:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5705 00000100:00080000:0.0:1788204250.628515:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204250.628524:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204250.628577:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187932928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204250.628607:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204250.628611:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204250.628613:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204250.628617:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5705 00000100:00080000:0.0:1788204250.628621:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204250.628631:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204250.628643:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187932928:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788204250.628646:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204250.628660:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187933056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204250.628670:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204250.628673:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204250.628675:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204250.628678:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5705 00000100:00080000:0.0:1788204250.628682:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204250.628686:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204250.628691:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204250.628701:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187933184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204250.628711:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204250.628714:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204250.628715:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788204250.628722:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204250.628723:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5705 00000100:00080000:0.0:1788204250.628729:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204250.628734:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204250.628740:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204250.628750:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187933312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204250.628759:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204250.628763:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204250.628764:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204250.628770:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5705 00000100:00100000:2.0:1788204250.628770:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204250/real 1788204250] req@ffff8e828793f480 x1875062187932928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204250.628777:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204250.628785:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204250.628790:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187932928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788204250.628791:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788204250.628804:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788204250.628808:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187933440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204250.628809:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204250.628815:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187932928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204250.628840:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187933056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204250.628858:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204250.628870:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204250/real 1788204250] req@ffff8e828793dc00 x1875062187933056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204250.628881:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187933056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204250.628889:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204250.628892:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204250.628894:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187933056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204250.628899:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187933184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204250.628913:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204250.628921:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204250/real 1788204250] req@ffff8e828793c000 x1875062187933184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204250.628928:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187933184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204250.628933:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204250.628935:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204250.628937:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187933184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204250.628939:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187933312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204250.628949:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204250.628954:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204250/real 1788204250] req@ffff8e828793ca80 x1875062187933312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204250.628960:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187933312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204250.628964:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204250.628966:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204250.628967:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187933312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204250.628970:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187933440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204250.628978:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204250.628983:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204250/real 1788204250] req@ffff8e828793c700 x1875062187933440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204250.628988:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187933440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204250.628996:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204250.628999:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204250.629002:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187933440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204251.332491:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204252.356523:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204253.380513:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204254.404527:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204255.428463:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204255.748438:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204255.748451:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204255.748457:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204255.748472:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5710 00000100:00080000:0.0:1788204255.748486:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204255.748496:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204255.748538:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187933568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204255.748567:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204255.748572:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204255.748575:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204255.748581:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5710 00000100:00080000:0.0:1788204255.748584:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204255.748589:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204255.748600:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187933568:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788204255.748627:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788204255.748671:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204255.748672:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187933696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204255.748688:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204255.748692:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204255.748694:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204255.748700:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5710 00000100:00080000:0.0:1788204255.748706:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204255.748713:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204255.748719:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204255.748721:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204255/real 1788204255] req@ffff8e828793f100 x1875062187933568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204255.748740:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187933824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204255.748753:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187933568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204255.748759:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204255.748767:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204255.748772:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204255.748776:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187933568:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204255.748777:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204255.748782:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204255.748789:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5710 00000100:00080000:0.0:1788204255.748795:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788204255.748803:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187933696:192.168.201.121@tcp:38:kworker.0 00000100:00080000:0.0:1788204255.748804:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204255.748816:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204255.748832:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187933952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204255.748844:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204255.748849:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204255.748854:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204255.748856:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204255.748863:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5710 00000100:00100000:1.0:1788204255.748865:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204255/real 1788204255] req@ffff8e828793e680 x1875062187933696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204255.748871:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204255.748881:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204255.748890:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187933696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788204255.748906:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788204255.748909:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204255.748916:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204255.748928:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187934080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204255.748928:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187933696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204255.748951:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187933824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204255.748983:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204255.748997:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204255/real 1788204255] req@ffff8e828793f800 x1875062187933824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204255.749011:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187933824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204255.749047:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204255.749051:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204255.749056:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187933824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204255.749072:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187933952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204255.749087:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204255.749096:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204255/real 1788204255] req@ffff8e828793ce00 x1875062187933952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204255.749103:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187933952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204255.749108:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204255.749110:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204255.749112:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187933952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788204255.749115:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187934080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204255.749125:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204255.749131:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204255/real 1788204255] req@ffff8e828793f100 x1875062187934080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204255.749136:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187934080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204255.749140:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204255.749142:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204255.749144:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187934080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204256.452501:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204257.478049:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204258.500498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204259.524519:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204260.548429:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204260.868441:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204260.868449:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204260.868452:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204260.868460:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5715 00000100:00080000:1.0:1788204260.868469:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204260.868474:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204260.868503:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187934208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204260.868524:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204260.868527:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204260.868528:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204260.868530:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5715 00000100:00080000:1.0:1788204260.868533:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204260.868537:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204260.868546:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204260.868560:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187934336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204260.868570:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788204260.868573:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187934208:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204260.868575:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204260.868579:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204260.868585:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5715 00000100:00080000:1.0:1788204260.868588:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204260.868592:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204260.868597:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204260.868610:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187934464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204260.868620:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204260.868623:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204260.868625:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204260.868630:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5715 00000100:00080000:1.0:1788204260.868634:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788204260.868636:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788204260.868641:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204260.868647:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204260.868658:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187934592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204260.868676:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788204260.868681:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204260/real 1788204260] req@ffff8e828d723800 x1875062187934208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204260.868682:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204260.868686:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204260.868694:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5715 00000100:00080000:1.0:1788204260.868701:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204260.868706:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204260.868711:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187934208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788204260.868713:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788204260.868727:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1788204260.868731:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187934720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204260.868735:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204260.868743:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187934208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204260.868770:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187934336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204260.868794:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204260.868807:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204260/real 1788204260] req@ffff8e828d721500 x1875062187934336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204260.868820:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187934336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204260.868828:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204260.868831:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204260.868834:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187934336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204260.868841:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187934464:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204260.868859:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204260.868870:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204260/real 1788204260] req@ffff8e828d720000 x1875062187934464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204260.868881:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187934464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204260.868889:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204260.868892:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204260.868895:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187934464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204260.868901:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187934592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204260.868943:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204260.868955:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204260/real 1788204260] req@ffff8e828d722680 x1875062187934592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204260.868966:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187934592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204260.868973:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204260.868976:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204260.868979:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187934592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204260.868985:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187934720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204260.869004:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204260.869015:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204260/real 1788204260] req@ffff8e828d720e00 x1875062187934720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204260.869027:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187934720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204260.869033:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204260.869035:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204260.869037:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187934720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204261.572488:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204262.596514:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204263.620570:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204264.644509:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204265.668474:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204265.988451:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204265.988465:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204265.988471:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204265.988483:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5721 00000100:00080000:1.0:1788204265.988496:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204265.988502:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204265.988544:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187934848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204265.988574:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204265.988578:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204265.988580:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204265.988584:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5721 00000100:00080000:1.0:1788204265.988588:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204265.988592:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204265.988604:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204265.988618:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187934976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204265.988625:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187934848:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204265.988638:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204265.988641:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204265.988642:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204265.988646:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5721 00000100:00080000:1.0:1788204265.988649:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204265.988652:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204265.988656:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204265.988665:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187935104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204265.988670:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204265.988673:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204265.988673:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204265.988676:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5721 00000100:00080000:1.0:1788204265.988684:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204265.988687:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204265.988691:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788204265.988695:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204265.988701:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187935232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204265.988708:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204265.988711:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204265.988712:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204265.988715:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5721 00000100:00080000:1.0:1788204265.988718:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204265.988720:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204265.988723:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204265.988729:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187935360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204265.988743:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204265/real 1788204265] req@ffff8e828d720e00 x1875062187934848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204265.988764:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187934848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204265.988775:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204265.988779:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204265.988784:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187934848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204265.988812:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187934976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204265.988830:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204265.988862:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204265/real 1788204265] req@ffff8e828d722680 x1875062187934976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204265.988872:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187934976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204265.988877:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204265.988879:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204265.988882:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187934976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204265.988885:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187935104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204265.988900:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204265.988907:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204265/real 1788204265] req@ffff8e828d720000 x1875062187935104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204265.988914:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187935104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204265.988919:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204265.988920:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204265.988922:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187935104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204265.988926:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187935232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204265.988944:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204265.988953:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204265/real 1788204265] req@ffff8e828d721500 x1875062187935232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204265.988965:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187935232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204265.988973:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204265.988976:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204265.988979:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187935232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204265.988984:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187935360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204265.988998:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204265.989005:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204265/real 1788204265] req@ffff8e828d723800 x1875062187935360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204265.989023:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187935360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204265.989029:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204265.989031:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204265.989033:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187935360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204266.692520:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204267.716470:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204268.740474:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204269.764508:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204270.788520:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204271.108432:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204271.108442:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204271.108445:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204271.108455:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5726 00000100:00080000:0.0:1788204271.108466:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204271.108473:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204271.108519:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187935488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204271.108548:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204271.108552:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204271.108553:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204271.108558:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5726 00000100:00080000:0.0:1788204271.108562:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204271.108566:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204271.108574:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204271.108584:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187935616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204271.108591:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204271.108594:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204271.108595:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204271.108597:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5726 00000100:00080000:0.0:1788204271.108599:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204271.108601:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204271.108604:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204271.108611:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187935744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204271.108617:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204271.108619:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204271.108620:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204271.108623:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5726 00000100:00080000:0.0:1788204271.108625:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204271.108629:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204271.108634:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204271.108650:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187935872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204271.108652:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187935488:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204271.108672:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204271.108675:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204271.108676:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204271.108681:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5726 00000100:00080000:0.0:1788204271.108683:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204271.108686:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204271.108692:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204271.108706:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187936000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204271.108777:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204271.108848:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204271/real 1788204271] req@ffff8e828793c700 x1875062187935488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204271.108871:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187935488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204271.108884:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204271.108889:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204271.108893:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187935488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204271.108901:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187935616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204271.108924:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204271.108937:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204271/real 1788204271] req@ffff8e828793ca80 x1875062187935616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204271.108949:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187935616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204271.108963:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204271.108967:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204271.108970:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187935616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204271.108977:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187935744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204271.108997:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204271.109013:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204271/real 1788204271] req@ffff8e828793c000 x1875062187935744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204271.109026:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187935744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204271.109034:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204271.109037:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204271.109040:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187935744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204271.109069:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187935872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204271.109090:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204271.109101:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204271/real 1788204271] req@ffff8e828793dc00 x1875062187935872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204271.109113:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187935872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204271.109120:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204271.109123:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204271.109127:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187935872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204271.109134:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187936000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204271.109153:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204271.109164:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204271/real 1788204271] req@ffff8e828793f480 x1875062187936000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204271.109176:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187936000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204271.109184:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204271.109187:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204271.109190:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187936000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204271.812519:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204272.836482:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204273.860477:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204274.884476:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204275.908462:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204276.228397:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204276.228408:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204276.228412:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204276.228422:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5731 00000100:00080000:0.0:1788204276.228432:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204276.228439:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204276.228475:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187936128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204276.228498:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204276.228501:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204276.228503:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204276.228508:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5731 00000100:00080000:0.0:1788204276.228511:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204276.228515:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204276.228524:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204276.228537:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187936256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204276.228548:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204276.228550:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187936128:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204276.228553:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204276.228557:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204276.228563:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5731 00000100:00080000:0.0:1788204276.228566:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204276.228570:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204276.228575:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204276.228588:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187936384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204276.228596:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204276.228598:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204276.228599:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204276.228603:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5731 00000100:00080000:0.0:1788204276.228606:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204276.228609:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204276.228612:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788204276.228617:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204276.228623:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187936512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204276.228629:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204276.228638:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204276.228640:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204276.228642:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5731 00000100:00080000:0.0:1788204276.228644:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204276.228646:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204276.228650:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204276.228676:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204276/real 1788204276] req@ffff8e828793c380 x1875062187936128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204276.228678:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187936640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204276.228708:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187936128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204276.228719:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204276.228724:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204276.228728:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187936128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204276.228749:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187936256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204276.228769:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204276.228781:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204276/real 1788204276] req@ffff8e828793e300 x1875062187936256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204276.228793:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187936256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204276.228801:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204276.228804:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204276.228807:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187936256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204276.228813:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187936384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204276.228829:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204276.228839:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204276/real 1788204276] req@ffff8e828793d500 x1875062187936384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204276.228851:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187936384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204276.228859:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204276.228862:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204276.228865:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187936384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204276.228871:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187936512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204276.228888:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204276.228897:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204276/real 1788204276] req@ffff8e828793fb80 x1875062187936512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204276.228909:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187936512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204276.228917:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204276.228920:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204276.228924:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187936512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204276.228929:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187936640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204276.228946:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204276.228955:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204276/real 1788204276] req@ffff8e828793d880 x1875062187936640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204276.228967:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187936640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204276.228974:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204276.228977:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204276.228981:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187936640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204276.932476:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204277.956477:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204278.980510:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204280.004522:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204281.028500:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204281.348433:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204281.348444:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204281.348449:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204281.348459:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5736 00000100:00080000:0.0:1788204281.348471:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204281.348478:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204281.348520:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187936768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204281.348552:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204281.348555:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204281.348557:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204281.348562:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5736 00000100:00080000:0.0:1788204281.348566:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204281.348580:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204281.348592:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204281.348605:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187936768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204281.348612:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187936896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204281.348623:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204281.348626:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204281.348628:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204281.348632:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5736 00000100:00080000:0.0:1788204281.348635:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204281.348639:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204281.348644:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204281.348655:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187937024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204281.348664:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204281.348667:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204281.348678:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204281.348683:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5736 00000100:00080000:0.0:1788204281.348686:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788204281.348690:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204281.348693:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204281.348699:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204281.348708:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187937152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204281.348720:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204281.348723:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204281.348724:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204281.348727:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5736 00000100:00080000:0.0:1788204281.348729:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204281.348732:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204281.348736:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204281.348744:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187937280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204281.348759:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204281/real 1788204281] req@ffff8e828793d880 x1875062187936768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204281.348781:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187936768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204281.348794:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204281.348798:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204281.348802:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187936768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204281.348829:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187936896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204281.348853:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204281.348872:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204281/real 1788204281] req@ffff8e828793fb80 x1875062187936896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204281.348885:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187936896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204281.348893:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204281.348896:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204281.348900:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187936896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204281.348906:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187937024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204281.348932:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204281.348942:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204281/real 1788204281] req@ffff8e828793d500 x1875062187937024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204281.348952:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187937024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204281.348959:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204281.348961:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204281.348965:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187937024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204281.348970:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187937152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204281.348989:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204281.348998:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204281/real 1788204281] req@ffff8e828793e300 x1875062187937152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204281.349011:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187937152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204281.349027:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204281.349033:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204281.349036:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187937152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204281.349043:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187937280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204281.349058:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204281.349065:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204281/real 1788204281] req@ffff8e828793c380 x1875062187937280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204281.349072:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187937280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204281.349077:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204281.349079:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204281.349081:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187937280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204282.052504:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204283.076570:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204284.100508:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204285.124486:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204286.148509:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204286.468437:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204286.468447:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204286.468456:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204286.468467:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5741 00000100:00080000:2.0:1788204286.468493:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204286.468501:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204286.468548:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187937408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204286.468569:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204286.468572:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204286.468574:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204286.468577:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5741 00000100:00080000:2.0:1788204286.468579:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204286.468582:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204286.468591:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204286.468599:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187937536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204286.468609:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187937408:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204286.468616:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204286.468619:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204286.468621:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204286.468623:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5741 00000100:00080000:2.0:1788204286.468625:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204286.468627:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204286.468630:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204286.468637:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187937664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204286.468641:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204286.468643:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204286.468643:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204286.468646:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5741 00000100:00080000:2.0:1788204286.468648:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204286.468651:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204286.468654:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204286.468660:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187937792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204286.468664:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204286.468666:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204286.468668:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204286.468670:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5741 00000100:00080000:2.0:1788204286.468672:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204286.468675:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204286.468677:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204286.468685:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187937920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204286.468713:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204286.468774:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204286/real 1788204286] req@ffff8e82894e6680 x1875062187937408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204286.468801:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187937408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204286.468814:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204286.468817:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204286.468820:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187937408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204286.468842:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187937536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204286.468859:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204286.468868:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204286/real 1788204286] req@ffff8e82894e7480 x1875062187937536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204286.468875:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187937536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204286.468881:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204286.468883:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204286.468885:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187937536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204286.468888:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187937664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204286.468898:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204286.468904:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204286/real 1788204286] req@ffff8e82894e6a00 x1875062187937664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204286.468911:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187937664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204286.468915:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204286.468917:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204286.468918:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187937664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204286.468922:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187937792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204286.468932:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204286.468937:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204286/real 1788204286] req@ffff8e82894e5880 x1875062187937792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204286.468944:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187937792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204286.468949:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204286.468950:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204286.468952:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187937792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204286.468955:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187937920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204286.468963:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204286.468968:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204286/real 1788204286] req@ffff8e82894e4000 x1875062187937920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204286.468973:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187937920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204286.468977:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204286.468979:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204286.468982:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187937920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204287.172557:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204288.196526:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204289.220514:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204290.244567:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204291.268528:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204291.588509:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204291.588517:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204291.588521:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204291.588529:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5746 00000100:00080000:0.0:1788204291.588537:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204291.588542:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204291.588575:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187938048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204291.588598:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204291.588602:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204291.588603:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204291.588610:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5746 00000100:00080000:0.0:1788204291.588612:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204291.588617:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204291.588627:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204291.588637:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187938048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204291.588640:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187938176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204291.588663:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204291.588668:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204291.588669:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204291.588676:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5746 00000100:00080000:0.0:1788204291.588681:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204291.588686:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204291.588693:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204291.588708:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187938304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204291.588720:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204291.588726:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204291.588729:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788204291.588731:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204291.588742:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5746 00000100:00080000:0.0:1788204291.588750:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204291.588755:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204291.588762:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204291.588775:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187938432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204291.588783:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204291.588785:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204291.588787:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204291.588790:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5746 00000100:00080000:0.0:1788204291.588792:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204291.588794:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204291.588797:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204291.588799:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204291/real 1788204291] req@ffff8e828793f480 x1875062187938048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204291.588810:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187938560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204291.588819:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187938048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204291.588829:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204291.588833:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204291.588839:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187938048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204291.588865:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187938176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204291.588883:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204291.588892:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204291/real 1788204291] req@ffff8e828793dc00 x1875062187938176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204291.588900:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187938176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204291.588906:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204291.588908:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204291.588911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187938176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204291.588915:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187938304:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204291.588927:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204291.588933:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204291/real 1788204291] req@ffff8e828793c000 x1875062187938304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204291.588940:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187938304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204291.588945:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204291.588947:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204291.588949:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187938304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204291.588953:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187938432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204291.588965:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204291.588972:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204291/real 1788204291] req@ffff8e828793ca80 x1875062187938432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204291.588978:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187938432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204291.588983:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204291.588986:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204291.588988:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187938432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204291.588992:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187938560:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204291.589004:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204291.589015:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204291/real 1788204291] req@ffff8e828793c700 x1875062187938560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204291.589030:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187938560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204291.589034:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204291.589036:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204291.589040:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187938560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204292.292484:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204293.316471:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204294.340505:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204295.364436:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204296.388479:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204296.708432:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204296.708442:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204296.708446:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204296.708455:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5751 00000100:00080000:2.0:1788204296.708470:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204296.708478:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204296.708512:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187938688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204296.708538:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204296.708542:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204296.708543:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204296.708547:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5751 00000100:00080000:2.0:1788204296.708550:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204296.708554:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204296.708563:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204296.708575:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187938816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204296.708579:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187938688:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204296.708596:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204296.708599:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204296.708601:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204296.708606:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5751 00000100:00080000:2.0:1788204296.708609:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204296.708614:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204296.708619:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204296.708631:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187938944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204296.708641:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204296.708644:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204296.708649:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788204296.708649:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204296.708658:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5751 00000100:00080000:2.0:1788204296.708663:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204296.708668:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204296.708673:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204296.708685:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187939072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204296.708694:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204296/real 1788204296] req@ffff8e82894e7800 x1875062187938688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204296.708697:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204296.708707:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204296.708710:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204296.708717:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5751 00000100:00100000:3.0:1788204296.708721:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187938688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204296.708723:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204296.708732:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:3.0:1788204296.708735:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1788204296.708739:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788204296.708741:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204296.708746:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187938688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204296.708755:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187939200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204296.708771:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187938816:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204296.708792:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204296.708802:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204296/real 1788204296] req@ffff8e82894e6300 x1875062187938816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204296.708812:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187938816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204296.708820:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204296.708823:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204296.708826:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187938816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204296.708830:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187938944:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204296.708844:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204296.708852:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204296/real 1788204296] req@ffff8e82894e4a80 x1875062187938944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204296.708861:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187938944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204296.708873:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204296.708876:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204296.708879:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187938944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204296.708884:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187939072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204296.708897:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204296.708905:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204296/real 1788204296] req@ffff8e82894e4e00 x1875062187939072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204296.708916:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187939072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204296.708923:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204296.708926:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204296.708928:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187939072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204296.708932:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187939200:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204296.708944:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204296.708952:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204296/real 1788204296] req@ffff8e82894e6d80 x1875062187939200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204296.708961:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187939200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204296.708968:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204296.708971:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204296.708974:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187939200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204297.412535:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204298.436501:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204299.460436:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204300.484520:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204301.508471:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204301.828417:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204301.828426:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204301.828430:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204301.828440:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5756 00000100:00080000:1.0:1788204301.828456:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204301.828463:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204301.828495:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187939328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204301.828513:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204301.828516:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204301.828517:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204301.828520:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5756 00000100:00080000:1.0:1788204301.828522:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204301.828525:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204301.828532:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204301.828545:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187939328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204301.828546:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187939456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204301.828566:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204301.828570:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204301.828573:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204301.828576:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5756 00000100:00080000:1.0:1788204301.828581:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204301.828585:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204301.828589:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204301.828599:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187939584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204301.828609:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204301.828612:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204301.828614:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204301.828618:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5756 00000100:00080000:1.0:1788204301.828621:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204301.828625:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204301.828630:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204301.828641:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187939712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204301.828644:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187939456:192.168.201.121@tcp:38:kworker.0 00000100:00080000:1.0:1788204301.828659:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204301.828663:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204301.828665:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204301.828669:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5756 00000100:00080000:1.0:1788204301.828673:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204301.828677:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187939584:192.168.201.121@tcp:38:kworker.0 00000100:00080000:1.0:1788204301.828679:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204301.828687:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204301.828698:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187939712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788204301.828700:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187939840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204301.828719:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187939840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204302.532442:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204303.556478:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204304.580473:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788204304.900573:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:2.0:1788204304.900613:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788204304.900629:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788204304.900667:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788204304.900675:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788204304.900682:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788204304.900689:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:3.0:1788204304.900843:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204301/real 1788204304] req@ffff8e828d720a80 x1875062187939840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204304.900865:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187939840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204304.900877:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204304.900882:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204304.900886:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187939840:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204304.900894:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204301/real 1788204304] req@ffff8e828d720380 x1875062187939712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204304.900906:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187939712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204304.900912:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204304.900915:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204304.900918:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187939712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204304.900925:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204301/real 1788204304] req@ffff8e828d722d80 x1875062187939456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204304.900935:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187939456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204304.900943:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204304.900946:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204304.900949:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187939456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204304.900955:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204301/real 1788204304] req@ffff8e828d721180 x1875062187939584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204304.900967:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187939584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204304.900973:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204304.900976:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204304.900979:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187939584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204304.900987:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204301/real 1788204304] req@ffff8e828d722300 x1875062187939328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204304.900996:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187939328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204304.901002:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204304.901005:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204304.901008:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187939328:192.168.201.121@tcp:250:kworker.0 00010000:00010000:0.0:1788204305.604519:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204306.628475:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204306.948439:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204306.948450:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204306.948454:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204306.948464:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5762 00000100:00080000:3.0:1788204306.948475:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204306.948483:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204306.948517:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187939968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204306.948549:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204306.948553:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204306.948554:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204306.948558:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5762 00000100:00080000:3.0:1788204306.948562:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204306.948566:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204306.948575:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204306.948591:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187939968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204306.948598:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187940096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204306.948610:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204306.948613:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204306.948615:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204306.948619:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5762 00000100:00080000:3.0:1788204306.948623:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204306.948627:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204306.948632:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204306.948646:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187940224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204306.948657:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204306.948661:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204306.948663:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204306.948667:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5762 00000800:00000100:2.0:1788204306.948672:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788204306.948674:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204306.948682:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204306.948688:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204306.948701:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187940352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204306.948717:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204306.948720:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204306.948722:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204306.948726:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5762 00000100:00080000:3.0:1788204306.948730:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788204306.948733:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204306/real 1788204306] req@ffff8e8289539500 x1875062187939968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204306.948737:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204306.948744:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204306.948760:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187940480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204306.948762:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187939968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204306.948786:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204306.948790:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204306.948794:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187939968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204306.948820:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187940096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204306.948851:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204306.948866:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204306/real 1788204306] req@ffff8e8289539f80 x1875062187940096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204306.948877:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187940096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204306.948884:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204306.948887:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204306.948890:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187940096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204306.948895:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187940224:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204306.948915:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204306.948924:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204306/real 1788204306] req@ffff8e828953a680 x1875062187940224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204306.948934:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187940224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204306.948941:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204306.948945:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204306.948948:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187940224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204306.948954:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187940352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204306.948971:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204306.948982:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204306/real 1788204306] req@ffff8e8289538380 x1875062187940352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204306.948991:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187940352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204306.948998:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204306.949001:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204306.949003:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187940352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204306.949007:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187940480:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204306.949024:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204306.949034:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204306/real 1788204306] req@ffff8e828953b100 x1875062187940480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204306.949045:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187940480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204306.949053:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204306.949057:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204306.949060:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187940480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204307.652439:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204308.676524:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204309.700450:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204310.724503:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204311.748465:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204312.068416:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204312.068426:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204312.068430:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204312.068439:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5767 00000100:00080000:3.0:1788204312.068449:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204312.068456:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204312.068489:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187940608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204312.068517:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204312.068520:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204312.068522:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204312.068525:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5767 00000100:00080000:3.0:1788204312.068528:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204312.068532:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204312.068540:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204312.068552:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187940736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204312.068562:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204312.068565:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204312.068567:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204312.068571:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5767 00000100:00080000:3.0:1788204312.068574:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204312.068579:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204312.068587:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204312.068597:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187940608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204312.068602:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187940864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204312.068625:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204312.068628:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204312.068631:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204312.068635:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5767 00000100:00080000:3.0:1788204312.068639:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204312.068643:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204312.068648:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204312.068662:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187940992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204312.068672:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204312.068675:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204312.068677:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204312.068681:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5767 00000100:00080000:3.0:1788204312.068685:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788204312.068689:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788204312.068694:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204312.068700:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204312.068713:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187941120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204312.068756:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204312/real 1788204312] req@ffff8e8289538a80 x1875062187940608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204312.068784:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187940608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204312.068795:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204312.068799:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204312.068801:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187940608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204312.068808:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187940736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204312.068828:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204312.068837:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204312/real 1788204312] req@ffff8e8289539880 x1875062187940736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204312.068845:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187940736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204312.068853:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204312.068856:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204312.068860:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187940736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204312.068889:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187940864:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204312.068908:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204312.068917:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204312/real 1788204312] req@ffff8e828953bb80 x1875062187940864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204312.068925:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187940864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204312.068930:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204312.068933:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204312.068934:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187940864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204312.068938:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187940992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204312.068951:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204312.068957:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204312/real 1788204312] req@ffff8e828953b100 x1875062187940992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204312.068964:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187940992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204312.068968:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204312.068970:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204312.068971:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187940992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204312.068974:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187941120:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204312.068984:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204312.068989:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204312/real 1788204312] req@ffff8e8289538380 x1875062187941120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204312.068995:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187941120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204312.068999:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204312.069001:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204312.069002:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187941120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204312.772445:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204313.796544:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204314.820448:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204315.844525:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204316.868571:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204317.188420:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204317.188436:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204317.188441:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204317.188452:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5772 00000100:00080000:2.0:1788204317.188463:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204317.188469:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204317.188507:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187941248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204317.188533:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204317.188536:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204317.188537:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204317.188540:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5772 00000100:00080000:2.0:1788204317.188542:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204317.188544:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204317.188555:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204317.188569:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187941376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204317.188572:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187941248:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204317.188586:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204317.188590:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204317.188592:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204317.188596:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5772 00000100:00080000:2.0:1788204317.188600:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204317.188604:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204317.188610:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204317.188622:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187941504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204317.188632:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204317.188636:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204317.188637:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204317.188640:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5772 00000100:00080000:2.0:1788204317.188644:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204317.188647:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204317.188651:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788204317.188653:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204317.188662:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187941632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204317.188669:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204317.188671:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204317.188673:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204317.188675:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5772 00000100:00080000:2.0:1788204317.188678:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204317.188680:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204317.188683:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204317.188691:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187941760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204317.188713:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204317/real 1788204317] req@ffff8e82894e5f80 x1875062187941248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204317.188733:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187941248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204317.188740:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204317.188743:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204317.188746:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187941248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204317.188768:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187941376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204317.188784:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204317.188792:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204317/real 1788204317] req@ffff8e82894e4380 x1875062187941376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204317.188799:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187941376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204317.188804:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204317.188805:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204317.188807:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187941376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204317.188810:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187941504:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204317.188820:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204317.188825:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204317/real 1788204317] req@ffff8e82894e5c00 x1875062187941504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204317.188831:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187941504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204317.188840:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204317.188842:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204317.188843:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187941504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204317.188846:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187941632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204317.188856:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204317.188861:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204317/real 1788204317] req@ffff8e82894e7100 x1875062187941632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204317.188866:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187941632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204317.188876:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204317.188878:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204317.188880:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187941632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204317.188883:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187941760:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204317.188894:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204317.188903:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204317/real 1788204317] req@ffff8e82894e6d80 x1875062187941760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204317.188913:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187941760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204317.188921:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204317.188924:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204317.188928:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187941760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204317.892482:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204318.916504:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204319.940534:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204320.964524:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204321.988493:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204322.308444:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204322.308459:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204322.308463:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204322.308474:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5777 00000100:00080000:2.0:1788204322.308486:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204322.308493:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204322.308533:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187941888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204322.308559:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204322.308563:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204322.308565:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204322.308568:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5777 00000100:00080000:2.0:1788204322.308570:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204322.308573:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204322.308582:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204322.308591:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187942016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204322.308598:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204322.308601:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204322.308603:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788204322.308605:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187941888:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204322.308607:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5777 00000100:00080000:2.0:1788204322.308611:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204322.308614:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204322.308617:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204322.308623:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187942144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204322.308628:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204322.308631:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204322.308632:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204322.308634:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5777 00000100:00080000:2.0:1788204322.308636:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204322.308639:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204322.308643:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204322.308651:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187942272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204322.308660:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204322.308663:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204322.308666:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204322.308670:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5777 00000100:00080000:2.0:1788204322.308674:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788204322.308677:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204322.308681:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204322.308688:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204322.308701:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187942400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204322.308733:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204322/real 1788204322] req@ffff8e82894e4e00 x1875062187941888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204322.308748:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187941888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204322.308756:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204322.308760:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204322.308763:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187941888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204322.308785:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187942016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204322.308799:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204322.308806:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204322/real 1788204322] req@ffff8e82894e4a80 x1875062187942016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204322.308814:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187942016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204322.308819:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204322.308821:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204322.308823:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187942016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204322.308827:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187942144:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204322.308836:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204322.308842:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204322/real 1788204322] req@ffff8e82894e6300 x1875062187942144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204322.308849:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187942144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204322.308853:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204322.308855:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204322.308856:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187942144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204322.308874:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187942272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204322.308889:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204322.308895:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204322/real 1788204322] req@ffff8e82894e7800 x1875062187942272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204322.308902:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187942272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204322.308907:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204322.308909:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204322.308911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187942272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204322.308914:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187942400:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204322.308928:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204322.308933:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204322/real 1788204322] req@ffff8e82894e4000 x1875062187942400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204322.308940:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187942400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204322.308945:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204322.308947:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204322.308949:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187942400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204323.012492:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204324.036474:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204325.060520:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204326.084537:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204327.108521:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204327.428505:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204327.428515:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204327.428521:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204327.428536:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5782 00000100:00080000:0.0:1788204327.428550:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204327.428559:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204327.428603:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187942528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204327.428633:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204327.428637:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204327.428639:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204327.428644:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5782 00000100:00080000:0.0:1788204327.428648:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204327.428653:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204327.428666:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204327.428681:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187942656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204327.428693:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204327.428698:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204327.428700:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204327.428705:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5782 00000100:00080000:0.0:1788204327.428709:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204327.428713:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204327.428720:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204327.428724:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187942528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204327.428735:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187942784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204327.428746:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204327.428750:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204327.428752:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204327.428757:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5782 00000100:00080000:0.0:1788204327.428760:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204327.428764:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204327.428783:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204327.428795:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187942912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204327.428805:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204327.428809:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204327.428811:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204327.428815:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5782 00000100:00080000:0.0:1788204327.428827:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204327.428831:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204327.428836:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788204327.428842:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204327.428853:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187943040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204327.428903:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204327/real 1788204327] req@ffff8e828793f100 x1875062187942528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204327.428927:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187942528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204327.428940:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204327.428945:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204327.428950:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187942528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204327.428959:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187942656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204327.428992:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204327.429008:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204327/real 1788204327] req@ffff8e828793ce00 x1875062187942656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204327.429020:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187942656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204327.429026:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204327.429029:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204327.429032:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187942656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204327.429060:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187942784:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204327.429078:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204327.429088:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204327/real 1788204327] req@ffff8e828793f800 x1875062187942784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204327.429105:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187942784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204327.429116:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204327.429121:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204327.429125:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187942784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204327.429133:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187942912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204327.429159:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204327.429175:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204327/real 1788204327] req@ffff8e828793e680 x1875062187942912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204327.429197:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187942912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204327.429207:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204327.429211:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204327.429215:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187942912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204327.429221:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187943040:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204327.429245:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204327.429258:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204327/real 1788204327] req@ffff8e828793c700 x1875062187943040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204327.429272:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187943040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204327.429280:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204327.429284:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204327.429288:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187943040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204328.132523:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204329.156499:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204330.180521:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204331.204508:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204332.228475:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204332.548415:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204332.548426:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204332.548431:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204332.548441:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5787 00000100:00080000:1.0:1788204332.548452:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204332.548460:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204332.548506:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187943168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204332.548536:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204332.548539:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204332.548540:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204332.548543:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5787 00000100:00080000:1.0:1788204332.548545:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204332.548548:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204332.548558:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204332.548567:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187943296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204332.548574:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204332.548576:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204332.548578:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204332.548579:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5787 00000100:00080000:1.0:1788204332.548581:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204332.548583:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204332.548586:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204332.548592:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187943168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204332.548597:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187943424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204332.548607:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204332.548611:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204332.548613:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204332.548617:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5787 00000100:00080000:1.0:1788204332.548621:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204332.548625:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204332.548631:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204332.548644:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187943552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204332.548653:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204332.548657:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204332.548659:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204332.548662:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5787 00000100:00080000:1.0:1788204332.548666:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788204332.548671:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788204332.548675:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204332.548693:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204332.548706:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187943680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204332.548747:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204332/real 1788204332] req@ffff8e828d722300 x1875062187943168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204332.548762:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187943168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204332.548774:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204332.548777:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204332.548781:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187943168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204332.548802:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187943296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204332.548819:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204332.548828:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204332/real 1788204332] req@ffff8e828d721180 x1875062187943296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204332.548835:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187943296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204332.548840:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204332.548842:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204332.548844:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187943296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204332.548847:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187943424:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204332.548856:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204332.548861:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204332/real 1788204332] req@ffff8e828d722d80 x1875062187943424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204332.548868:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187943424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204332.548873:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204332.548874:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204332.548876:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187943424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204332.548879:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187943552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204332.548890:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204332.548895:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204332/real 1788204332] req@ffff8e828d720380 x1875062187943552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204332.548902:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187943552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204332.548906:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204332.548907:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204332.548909:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187943552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204332.548912:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187943680:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204332.548920:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204332.548925:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204332/real 1788204332] req@ffff8e828d720a80 x1875062187943680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204332.548930:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187943680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204332.548934:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204332.548936:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204332.548938:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187943680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204333.252497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204334.276500:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204335.300478:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204336.324506:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204337.348514:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204337.668423:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204337.668433:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204337.668438:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204337.668449:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5792 00000100:00080000:1.0:1788204337.668461:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204337.668468:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204337.668510:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187943808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204337.668539:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204337.668542:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204337.668543:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204337.668546:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5792 00000100:00080000:1.0:1788204337.668548:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204337.668551:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204337.668561:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204337.668570:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187943936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204337.668577:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204337.668579:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204337.668581:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204337.668585:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5792 00000100:00100000:3.0:1788204337.668585:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187943808:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204337.668589:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204337.668606:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204337.668609:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204337.668618:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187944064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204337.668624:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204337.668626:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204337.668627:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204337.668630:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5792 00000100:00080000:1.0:1788204337.668632:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204337.668635:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204337.668638:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204337.668647:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187944192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204337.668653:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204337.668655:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204337.668656:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204337.668659:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5792 00000100:00080000:1.0:1788204337.668667:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204337.668670:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204337.668673:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788204337.668679:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204337.668681:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187944320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204337.668726:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204337/real 1788204337] req@ffff8e828d723800 x1875062187943808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204337.668746:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187943808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204337.668757:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204337.668761:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204337.668765:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187943808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204337.668792:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187943936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204337.668810:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204337.668818:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204337/real 1788204337] req@ffff8e828d721500 x1875062187943936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204337.668826:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187943936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204337.668832:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204337.668834:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204337.668836:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187943936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204337.668840:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187944064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204337.668851:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204337.668857:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204337/real 1788204337] req@ffff8e828d720000 x1875062187944064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204337.668864:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187944064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204337.668868:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204337.668870:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204337.668871:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187944064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204337.668874:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187944192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204337.668885:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204337.668890:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204337/real 1788204337] req@ffff8e828d722680 x1875062187944192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204337.668896:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187944192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204337.668901:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204337.668903:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204337.668904:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187944192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204337.668907:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187944320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204337.668915:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204337.668920:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204337/real 1788204337] req@ffff8e828d720e00 x1875062187944320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204337.668927:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187944320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204337.668931:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204337.668933:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204337.668934:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187944320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204338.372560:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204339.396500:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204340.420494:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204341.444458:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204342.468493:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204342.788452:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204342.788464:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204342.788469:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204342.788481:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5797 00000100:00080000:0.0:1788204342.788493:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204342.788500:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204342.788559:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187944448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204342.788584:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204342.788587:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204342.788588:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204342.788591:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5797 00000100:00080000:0.0:1788204342.788594:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204342.788597:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204342.788606:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204342.788622:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187944576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204342.788629:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204342.788632:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204342.788635:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788204342.788637:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187944448:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204342.788639:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5797 00000100:00080000:0.0:1788204342.788644:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204342.788647:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204342.788650:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204342.788657:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187944704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204342.788663:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204342.788665:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204342.788666:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204342.788668:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5797 00000100:00080000:0.0:1788204342.788670:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204342.788673:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204342.788676:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204342.788683:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187944832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204342.788687:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204342.788690:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204342.788691:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204342.788693:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5797 00000100:00080000:0.0:1788204342.788696:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204342.788698:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204342.788700:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204342.788707:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187944960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204342.788719:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204342.788770:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204342/real 1788204342] req@ffff8e828793ca80 x1875062187944448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204342.788786:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187944448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204342.788795:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204342.788798:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204342.788801:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187944448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204342.788823:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187944576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204342.788839:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204342.788847:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204342/real 1788204342] req@ffff8e828793c000 x1875062187944576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204342.788854:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187944576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204342.788863:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204342.788866:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204342.788868:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187944576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204342.788871:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187944704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204342.788891:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204342.788901:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204342/real 1788204342] req@ffff8e828793dc00 x1875062187944704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204342.788913:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187944704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204342.788921:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204342.788924:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204342.788927:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187944704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204342.788933:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187944832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204342.788954:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204342.788965:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204342/real 1788204342] req@ffff8e828793f480 x1875062187944832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204342.788975:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187944832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204342.788980:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204342.788982:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204342.788985:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187944832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204342.788989:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187944960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204342.789002:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204342.789008:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204342/real 1788204342] req@ffff8e828793c380 x1875062187944960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204342.789014:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187944960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204342.789018:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204342.789020:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204342.789021:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187944960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204343.492505:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204344.516513:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204345.540517:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204346.564519:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204347.588517:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204347.908447:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204347.908463:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204347.908468:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204347.908480:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5802 00000100:00080000:0.0:1788204347.908495:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204347.908504:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204347.908557:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187945088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204347.908589:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204347.908593:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204347.908595:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204347.908600:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5802 00000100:00080000:0.0:1788204347.908605:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204347.908610:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204347.908622:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204347.908631:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187945088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204347.908636:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187945216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204347.908645:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204347.908648:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204347.908649:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204347.908653:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5802 00000100:00080000:0.0:1788204347.908656:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204347.908658:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204347.908662:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204347.908671:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187945344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204347.908677:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204347.908679:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204347.908681:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204347.908683:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5802 00000100:00080000:0.0:1788204347.908685:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204347.908687:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204347.908690:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204347.908695:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187945472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204347.908700:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204347.908702:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204347.908703:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788204347.908707:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204347.908714:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5802 00000100:00080000:0.0:1788204347.908717:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204347.908720:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204347.908723:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204347.908734:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187945600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204347.908761:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204347/real 1788204347] req@ffff8e828793e300 x1875062187945088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204347.908776:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187945088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204347.908783:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204347.908786:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204347.908789:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187945088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204347.908810:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187945216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204347.908825:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204347.908833:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204347/real 1788204347] req@ffff8e828793d500 x1875062187945216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204347.908840:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187945216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204347.908846:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204347.908848:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204347.908850:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187945216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204347.908853:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187945344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204347.908862:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204347.908868:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204347/real 1788204347] req@ffff8e828793fb80 x1875062187945344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204347.908874:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187945344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204347.908879:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204347.908880:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204347.908882:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187945344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204347.908885:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187945472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204347.908894:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204347.908899:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204347/real 1788204347] req@ffff8e828793d880 x1875062187945472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204347.908905:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187945472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204347.908910:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204347.908913:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204347.908914:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187945472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204347.908917:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187945600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204347.908926:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204347.908931:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204347/real 1788204347] req@ffff8e828793c380 x1875062187945600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204347.908936:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187945600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204347.908940:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204347.908942:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204347.908943:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187945600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204348.612533:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204349.636489:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204350.660548:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204351.684524:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204352.708502:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204353.028448:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204353.028463:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204353.028468:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204353.028482:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5808 00000100:00080000:0.0:1788204353.028495:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204353.028504:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204353.028556:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187945728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204353.028598:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204353.028602:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204353.028604:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204353.028608:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5808 00000100:00080000:0.0:1788204353.028612:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204353.028616:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204353.028627:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204353.028644:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187945856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204353.028653:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187945728:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204353.028659:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204353.028663:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204353.028665:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204353.028670:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5808 00000100:00080000:0.0:1788204353.028674:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204353.028679:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204353.028684:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204353.028696:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187945984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204353.028705:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204353.028708:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204353.028710:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204353.028715:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5808 00000100:00080000:0.0:1788204353.028719:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204353.028723:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:3.0:1788204353.028727:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788204353.028729:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204353.028742:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187946112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204353.028752:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204353.028755:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204353.028757:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204353.028761:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5808 00000100:00080000:0.0:1788204353.028765:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204353.028769:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204353.028774:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204353.028787:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187946240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204353.028788:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204353/real 1788204353] req@ffff8e828793f480 x1875062187945728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204353.028823:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187945728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204353.028832:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204353.028835:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204353.028838:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187945728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204353.028860:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187945856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204353.028875:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204353.028889:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204353/real 1788204353] req@ffff8e828793dc00 x1875062187945856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204353.028897:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187945856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204353.028904:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204353.028907:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204353.028910:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187945856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204353.028916:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187945984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204353.028936:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204353.028947:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204353/real 1788204353] req@ffff8e828793c000 x1875062187945984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204353.028957:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187945984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204353.028963:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204353.028966:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204353.028969:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187945984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204353.028974:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187946112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204353.029005:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204353.029016:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204353/real 1788204353] req@ffff8e828793ca80 x1875062187946112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204353.029026:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187946112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204353.029032:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204353.029035:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204353.029038:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187946112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204353.029042:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187946240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204353.029060:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204353.029078:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204353/real 1788204353] req@ffff8e828793c700 x1875062187946240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204353.029090:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187946240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204353.029095:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204353.029097:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204353.029099:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187946240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204353.732439:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204354.756487:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204355.780525:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204356.804463:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204357.828527:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204358.148459:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204358.148474:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204358.148479:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204358.148493:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5813 00000100:00080000:1.0:1788204358.148508:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204358.148517:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204358.148568:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187946368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204358.148605:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204358.148610:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204358.148611:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204358.148616:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5813 00000100:00080000:1.0:1788204358.148621:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204358.148626:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204358.148636:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204358.148653:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187946496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204358.148662:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187946368:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204358.148667:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204358.148671:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204358.148681:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204358.148686:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5813 00000100:00080000:1.0:1788204358.148690:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204358.148694:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204358.148699:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204358.148713:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187946624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204358.148721:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204358.148725:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204358.148727:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204358.148731:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5813 00000100:00080000:1.0:1788204358.148735:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204358.148739:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204358.148744:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788204358.148750:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204358.148755:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187946752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204358.148770:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204358.148773:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204358.148775:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204358.148779:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5813 00000100:00080000:1.0:1788204358.148782:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204358.148785:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204358.148789:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204358.148800:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187946880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204358.148809:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204358/real 1788204358] req@ffff8e828d720e00 x1875062187946368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204358.148836:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187946368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204358.148851:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204358.148855:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204358.148860:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187946368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204358.148890:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187946496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204358.148912:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204358.148923:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204358/real 1788204358] req@ffff8e828d722680 x1875062187946496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204358.148931:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187946496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204358.148937:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204358.148939:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204358.148941:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187946496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204358.148945:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187946624:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204358.148956:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204358.148962:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204358/real 1788204358] req@ffff8e828d720000 x1875062187946624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204358.148969:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187946624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204358.148973:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204358.148975:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204358.148977:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187946624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204358.148980:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187946752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204358.148991:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204358.148996:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204358/real 1788204358] req@ffff8e828d721500 x1875062187946752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204358.149003:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187946752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204358.149008:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204358.149009:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204358.149011:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187946752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204358.149014:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187946880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204358.149023:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204358.149027:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204358/real 1788204358] req@ffff8e828d723800 x1875062187946880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204358.149033:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187946880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204358.149037:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204358.149038:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204358.149040:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187946880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204358.852501:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204359.876479:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204360.900459:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204361.924481:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204362.948534:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204363.268443:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204363.268457:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204363.268469:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204363.268483:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5818 00000100:00080000:3.0:1788204363.268496:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204363.268503:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204363.268542:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187947008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204363.268574:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204363.268578:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204363.268580:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204363.268585:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5818 00000100:00080000:3.0:1788204363.268589:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204363.268594:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204363.268606:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204363.268622:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187947136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204363.268634:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204363.268638:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204363.268640:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788204363.268645:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187947008:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204363.268646:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5818 00000100:00080000:3.0:1788204363.268652:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204363.268657:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204363.268661:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204363.268671:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187947264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204363.268682:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204363.268685:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204363.268686:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204363.268689:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5818 00000100:00080000:3.0:1788204363.268691:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204363.268693:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204363.268697:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204363.268708:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187947392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204363.268717:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204363.268721:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204363.268723:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204363.268730:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5818 00000800:00000100:2.0:1788204363.268731:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788204363.268738:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204363.268744:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204363.268788:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204363/real 1788204363] req@ffff8e828953a680 x1875062187947008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204363.268810:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187947008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788204363.268812:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788204363.268825:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788204363.268828:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187947520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204363.268833:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204363.268842:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187947008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204363.268870:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187947136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204363.268893:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204363.268907:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204363/real 1788204363] req@ffff8e8289539f80 x1875062187947136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204363.268920:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187947136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204363.268929:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204363.268932:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204363.268936:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187947136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204363.268942:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187947264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204363.268964:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204363.268977:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204363/real 1788204363] req@ffff8e8289539500 x1875062187947264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204363.268991:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187947264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204363.269005:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204363.269009:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204363.269013:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187947264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204363.269020:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187947392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204363.269042:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204363.269056:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204363/real 1788204363] req@ffff8e8289538380 x1875062187947392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204363.269070:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187947392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204363.269078:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204363.269083:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204363.269086:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187947392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204363.269093:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187947520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204363.269113:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204363.269125:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204363/real 1788204363] req@ffff8e828953b100 x1875062187947520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204363.269136:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187947520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204363.269167:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204363.269171:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204363.269175:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187947520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204363.972502:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204364.996524:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204366.020503:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204367.044522:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204368.068510:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204368.388466:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204368.388476:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204368.388480:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204368.388495:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5823 00000100:00080000:0.0:1788204368.388504:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204368.388511:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204368.388541:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187947648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204368.388563:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204368.388565:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204368.388567:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204368.388570:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5823 00000100:00080000:0.0:1788204368.388571:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204368.388573:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204368.388580:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204368.388589:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187947776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204368.388595:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204368.388597:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204368.388598:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204368.388601:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5823 00000100:00080000:0.0:1788204368.388602:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204368.388606:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204368.388607:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187947648:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788204368.388610:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204368.388620:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187947904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204368.388626:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204368.388628:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204368.388629:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204368.388631:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5823 00000100:00080000:0.0:1788204368.388633:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204368.388635:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204368.388638:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204368.388644:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187948032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204368.388650:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204368.388652:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204368.388653:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204368.388656:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5823 00000100:00080000:0.0:1788204368.388664:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204368.388667:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204368.388673:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204368.388685:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187948160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204368.388696:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187947776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204368.388736:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187947904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204368.388771:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187948032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204368.388784:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187948160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204369.092482:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204370.116521:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204371.140433:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788204371.460674:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788204371.460732:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788204371.460755:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204371.460806:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204371.460818:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204371.460826:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204371.460833:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:2.0:1788204371.463104:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204368/real 1788204371] req@ffff8e828793ce00 x1875062187947904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204371.463151:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187947904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204371.463165:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204371.463172:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204371.463176:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187947904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204371.463186:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204368/real 1788204371] req@ffff8e828793f100 x1875062187948032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204371.463199:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187948032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204371.463205:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204371.463208:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204371.463210:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187948032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204371.463216:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204368/real 1788204371] req@ffff8e828793c700 x1875062187948160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204371.463225:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187948160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204371.463229:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204371.463231:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204371.463233:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187948160:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204371.463238:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204368/real 1788204371] req@ffff8e828793e680 x1875062187947648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204371.463248:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187947648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204371.463255:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204371.463258:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204371.463261:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187947648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204371.463269:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204368/real 1788204371] req@ffff8e828793f800 x1875062187947776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204371.463280:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187947776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204371.463288:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204371.463291:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204371.463294:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187947776:192.168.201.121@tcp:38:kworker.0 00010000:00010000:1.0:1788204372.164526:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204373.188512:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204373.508414:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204373.508426:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204373.508430:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204373.508441:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5828 00000100:00080000:0.0:1788204373.508452:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204373.508458:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204373.508493:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187948288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204373.508521:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204373.508532:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204373.508534:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204373.508539:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5828 00000100:00080000:0.0:1788204373.508543:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204373.508548:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204373.508559:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204373.508574:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187948416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204373.508585:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788204373.508585:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187948288:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204373.508592:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204373.508595:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204373.508599:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5828 00000100:00080000:0.0:1788204373.508602:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204373.508607:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204373.508612:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204373.508623:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187948544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204373.508632:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204373.508635:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204373.508637:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204373.508641:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5828 00000100:00080000:0.0:1788204373.508645:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204373.508649:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204373.508655:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204373.508667:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187948672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204373.508676:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204373.508678:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204373.508696:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204373.508698:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204373.508704:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5828 00000100:00080000:0.0:1788204373.508708:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204373.508712:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204373.508718:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204373.508731:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187948800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204373.508740:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204373/real 1788204373] req@ffff8e828793ca80 x1875062187948288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204373.508766:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187948288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204373.508781:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204373.508786:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204373.508792:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187948288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204373.508825:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187948416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204373.508852:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204373.508867:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204373/real 1788204373] req@ffff8e828793c000 x1875062187948416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204373.508880:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187948416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204373.508889:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204373.508893:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204373.508896:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187948416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204373.508902:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187948544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204373.508923:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204373.508935:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204373/real 1788204373] req@ffff8e828793dc00 x1875062187948544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204373.508949:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187948544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204373.508957:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204373.508960:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204373.508963:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187948544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204373.508984:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187948672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204373.509009:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204373.509022:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204373/real 1788204373] req@ffff8e828793f480 x1875062187948672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204373.509034:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187948672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204373.509042:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204373.509045:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204373.509048:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187948672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204373.509054:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187948800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204373.509073:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204373.509083:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204373/real 1788204373] req@ffff8e828793c380 x1875062187948800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204373.509096:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187948800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204373.509110:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204373.509113:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204373.509116:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187948800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204374.212476:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204375.236477:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204376.260471:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204377.284487:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204378.308687:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204378.628453:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204378.628464:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204378.628469:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204378.628480:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5833 00000100:00080000:3.0:1788204378.628493:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204378.628500:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204378.628541:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187948928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204378.628563:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204378.628566:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204378.628567:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204378.628569:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5833 00000100:00080000:3.0:1788204378.628571:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204378.628574:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204378.628583:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204378.628592:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187949056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204378.628599:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788204378.628603:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187948928:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204378.628603:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204378.628605:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204378.628611:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5833 00000100:00080000:3.0:1788204378.628615:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204378.628617:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204378.628620:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204378.628628:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187949184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204378.628633:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204378.628635:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204378.628637:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204378.628639:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5833 00000100:00080000:3.0:1788204378.628641:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204378.628643:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204378.628646:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204378.628654:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187949312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204378.628659:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204378.628661:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204378.628662:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204378.628665:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5833 00000100:00080000:3.0:1788204378.628667:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204378.628669:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204378.628671:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788204378.628676:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204378.628681:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187949440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204378.628729:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204378/real 1788204378] req@ffff8e828953bb80 x1875062187948928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204378.628748:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187948928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204378.628760:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204378.628764:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204378.628768:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187948928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204378.628796:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187949056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204378.628819:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204378.628832:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204378/real 1788204378] req@ffff8e8289539880 x1875062187949056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204378.628843:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187949056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204378.628850:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204378.628860:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204378.628863:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187949056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204378.628870:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187949184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204378.628902:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204378.628914:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204378/real 1788204378] req@ffff8e8289538a80 x1875062187949184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204378.628926:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187949184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204378.628933:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204378.628936:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204378.628939:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187949184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204378.628945:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187949312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204378.628965:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204378.628976:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204378/real 1788204378] req@ffff8e828953b100 x1875062187949312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204378.628988:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187949312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204378.628994:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204378.628997:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204378.628999:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187949312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204378.629003:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187949440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204378.629017:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204378.629023:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204378/real 1788204378] req@ffff8e8289538380 x1875062187949440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204378.629030:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187949440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204378.629035:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204378.629037:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204378.629039:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187949440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204379.332503:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204380.356485:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204381.380477:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204382.404499:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204383.428631:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204383.748420:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204383.748430:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204383.748434:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204383.748445:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5838 00000100:00080000:3.0:1788204383.748455:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204383.748462:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204383.748498:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187949568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204383.748528:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204383.748532:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204383.748534:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204383.748538:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5838 00000100:00080000:3.0:1788204383.748542:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204383.748546:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204383.748554:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204383.748570:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187949696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204383.748581:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187949568:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204383.748584:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204383.748589:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204383.748591:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204383.748596:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5838 00000100:00080000:3.0:1788204383.748604:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204383.748608:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204383.748615:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204383.748627:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187949824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204383.748637:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204383.748641:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204383.748642:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788204383.748646:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788204383.748647:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5838 00000100:00080000:3.0:1788204383.748653:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204383.748657:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204383.748663:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204383.748675:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187949952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204383.748685:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204383.748688:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204383.748690:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788204383.748693:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204383/real 1788204383] req@ffff8e8289539500 x1875062187949568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204383.748697:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5838 00000100:00080000:3.0:1788204383.748704:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204383.748709:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204383.748714:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204383.748715:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187949568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204383.748726:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788204383.748727:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187950080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204383.748731:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204383.748738:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187949568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204383.748759:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187949696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204383.748776:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204383.748786:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204383/real 1788204383] req@ffff8e8289539f80 x1875062187949696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204383.748809:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187949696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204383.748824:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204383.748827:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204383.748831:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187949696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204383.748837:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187949824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204383.748852:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204383.748859:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204383/real 1788204383] req@ffff8e828953a680 x1875062187949824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204383.748866:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187949824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204383.748871:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204383.748873:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204383.748874:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187949824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204383.748877:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187949952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204383.748887:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204383.748892:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204383/real 1788204383] req@ffff8e8289538380 x1875062187949952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204383.748899:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187949952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204383.748903:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204383.748904:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204383.748907:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187949952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204383.748910:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187950080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204383.748919:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204383.748923:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204383/real 1788204383] req@ffff8e828953b100 x1875062187950080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204383.748929:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187950080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204383.748934:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204383.748935:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204383.748937:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187950080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204384.452465:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204385.476499:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204386.500492:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204387.524563:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204388.548524:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204388.868417:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204388.868426:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204388.868429:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204388.868437:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5843 00000100:00080000:2.0:1788204388.868445:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204388.868451:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204388.868485:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187950208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204388.868517:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204388.868520:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204388.868521:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204388.868524:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5843 00000100:00080000:2.0:1788204388.868527:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204388.868532:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204388.868540:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204388.868548:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187950208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204388.868554:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187950336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204388.868566:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204388.868569:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204388.868571:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204388.868575:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5843 00000100:00080000:2.0:1788204388.868578:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204388.868582:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204388.868587:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204388.868599:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187950464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204388.868608:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788204388.868613:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204388.868614:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204388.868618:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204388.868622:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5843 00000100:00080000:2.0:1788204388.868625:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204388.868630:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204388.868636:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204388.868648:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187950592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204388.868657:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204388.868660:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204388/real 1788204388] req@ffff8e82894e5880 x1875062187950208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204388.868662:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204388.868665:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204388.868672:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5843 00000100:00080000:2.0:1788204388.868678:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204388.868681:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204388.868685:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204388.868694:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187950208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204388.868696:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187950720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204388.868724:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204388.868728:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204388.868730:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187950208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204388.868759:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187950336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204388.868783:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204388.868796:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204388/real 1788204388] req@ffff8e82894e6a00 x1875062187950336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204388.868808:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187950336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204388.868816:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204388.868819:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204388.868823:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187950336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204388.868827:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187950464:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204388.868846:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204388.868857:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204388/real 1788204388] req@ffff8e82894e7480 x1875062187950464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204388.868869:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187950464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204388.868888:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204388.868892:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204388.868896:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187950464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204388.868902:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187950592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204388.868922:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204388.868939:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204388/real 1788204388] req@ffff8e82894e6680 x1875062187950592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204388.868951:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187950592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204388.868958:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204388.868962:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204388.868965:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187950592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204388.868971:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187950720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204388.868992:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204388.869000:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204388/real 1788204388] req@ffff8e82894e4000 x1875062187950720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204388.869007:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187950720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204388.869012:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204388.869014:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204388.869016:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187950720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204389.572497:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204390.596468:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204391.620487:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204392.644437:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204393.668477:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204393.988463:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204393.988473:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204393.988477:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204393.988487:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5849 00000100:00080000:1.0:1788204393.988498:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204393.988504:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204393.988552:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187950848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204393.988576:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204393.988579:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204393.988581:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204393.988585:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5849 00000100:00080000:1.0:1788204393.988595:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204393.988601:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204393.988610:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204393.988635:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187950976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204393.988641:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187950848:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204393.988650:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204393.988654:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204393.988656:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204393.988661:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5849 00000100:00080000:1.0:1788204393.988664:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204393.988668:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204393.988674:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204393.988685:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187951104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204393.988694:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204393.988697:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204393.988699:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204393.988702:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5849 00000100:00080000:1.0:1788204393.988706:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788204393.988708:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788204393.988713:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204393.988719:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204393.988730:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187951232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204393.988739:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204393.988741:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204393.988743:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204393.988747:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5849 00000100:00080000:1.0:1788204393.988750:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204393.988752:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204393/real 1788204393] req@ffff8e828d720a80 x1875062187950848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204393.988756:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204393.988764:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204393.988777:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187950848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204393.988779:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187951360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204393.988800:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204393.988807:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204393.988811:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187950848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204393.988835:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187950976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204393.988853:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204393.988865:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204393/real 1788204393] req@ffff8e828d720380 x1875062187950976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204393.988877:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187950976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204393.988885:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204393.988888:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204393.988891:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187950976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204393.988897:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187951104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204393.988917:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204393.988928:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204393/real 1788204393] req@ffff8e828d722d80 x1875062187951104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204393.988938:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187951104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204393.988944:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204393.988945:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204393.988947:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187951104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204393.988951:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187951232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204393.988965:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204393.988971:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204393/real 1788204393] req@ffff8e828d721180 x1875062187951232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204393.988978:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187951232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204393.988982:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204393.988983:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204393.988985:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187951232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204393.988987:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187951360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204393.988996:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204393.989001:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204393/real 1788204393] req@ffff8e828d722300 x1875062187951360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204393.989007:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187951360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204393.989012:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204393.989013:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204393.989015:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187951360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204394.692465:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204395.716502:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204396.740530:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204397.764495:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204398.788499:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204399.108478:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204399.108487:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204399.108492:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204399.108503:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5854 00000100:00080000:3.0:1788204399.108514:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204399.108522:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204399.108581:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187951488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204399.108609:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204399.108613:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204399.108615:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204399.108620:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5854 00000100:00080000:3.0:1788204399.108625:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788204399.108645:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187951488:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204399.108645:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204399.108660:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204399.108678:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187951616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204399.108690:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204399.108695:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204399.108697:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204399.108707:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5854 00000100:00080000:3.0:1788204399.108717:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204399.108722:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204399.108727:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204399.108740:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187951744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204399.108748:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204399.108790:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204399/real 1788204399] req@ffff8e8289538a80 x1875062187951488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204399.108805:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187951488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204399.108812:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204399.108816:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204399.108819:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187951488:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204399.108835:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204399.108839:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204399.108842:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204399.108847:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5854 00000100:00080000:3.0:1788204399.108852:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204399.108857:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204399.108863:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204399.108864:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187951616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204399.108898:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204399.108911:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204399/real 1788204399] req@ffff8e8289539880 x1875062187951616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204399.108926:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187951616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204399.108936:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204399.108940:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204399.108944:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187951616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204399.108949:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187951744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204399.108969:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204399.108981:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204399/real 1788204399] req@ffff8e828953bb80 x1875062187951744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204399.108994:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187951744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204399.109002:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204399.109006:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204399.109009:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187951872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204399.109010:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187951744:192.168.201.121@tcp:38:kworker.0 00000100:00080000:3.0:1788204399.109030:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788204399.109035:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187951872:192.168.201.121@tcp:8:kworker.0 00000100:00080000:3.0:1788204399.109036:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204399.109041:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204399.109049:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5854 00000800:00000100:2.0:1788204399.109075:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204399.109091:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204399/real 1788204399] req@ffff8e8289538a80 x1875062187951872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204399.109101:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204399.109110:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204399.109112:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187951872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788204399.109120:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788204399.109136:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788204399.109140:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187952000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204399.109143:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204399.109153:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187951872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204399.109173:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187952000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204399.109195:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204399.109206:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204399/real 1788204399] req@ffff8e828953b100 x1875062187952000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204399.109232:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187952000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204399.109240:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204399.109244:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204399.109247:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187952000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204399.812454:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204400.836494:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204401.860511:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204402.884514:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204403.908511:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204404.228480:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204404.228490:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204404.228495:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204404.228506:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5859 00000100:00080000:3.0:1788204404.228531:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204404.228539:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204404.228578:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187952128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204404.228610:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204404.228615:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204404.228617:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204404.228621:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5859 00000100:00080000:3.0:1788204404.228625:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204404.228629:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204404.228640:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204404.228658:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187952256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204404.228661:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187952128:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204404.228677:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204404.228682:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204404.228684:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204404.228689:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5859 00000100:00080000:3.0:1788204404.228694:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204404.228699:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204404.228705:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204404.228719:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187952384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204404.228721:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788204404.228733:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204404.228738:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204404.228740:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204404.228745:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5859 00000100:00080000:3.0:1788204404.228749:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204404.228754:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204404.228759:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204404.228770:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204404/real 1788204404] req@ffff8e8289538380 x1875062187952128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204404.228771:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187952512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204404.228798:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204404.228801:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204404.228802:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204404.228805:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5859 00000100:00080000:3.0:1788204404.228809:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788204404.228813:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187952128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204404.228815:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204404.228824:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788204404.228833:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204404.228842:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204404.228842:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187952640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204404.228851:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187952128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204404.228885:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187952256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204404.228911:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204404.228923:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204404/real 1788204404] req@ffff8e828953a680 x1875062187952256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204404.228932:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187952256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204404.228938:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204404.228940:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204404.228942:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187952256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204404.228946:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187952384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204404.228958:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204404.228964:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204404/real 1788204404] req@ffff8e8289539f80 x1875062187952384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204404.228972:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187952384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204404.228977:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204404.228979:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204404.228980:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187952384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204404.228984:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187952512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204404.228995:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204404.229000:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204404/real 1788204404] req@ffff8e8289539500 x1875062187952512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204404.229007:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187952512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204404.229012:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204404.229014:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204404.229017:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187952512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204404.229020:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187952640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204404.229029:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204404.229034:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204404/real 1788204404] req@ffff8e828953b100 x1875062187952640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204404.229040:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187952640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204404.229045:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204404.229047:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204404.229049:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187952640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204404.932529:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204405.956501:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204406.980467:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204408.004489:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204409.028544:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204409.348456:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204409.348471:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204409.348477:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204409.348492:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5864 00000100:00080000:0.0:1788204409.348506:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204409.348525:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204409.348577:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187952768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204409.348615:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204409.348620:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204409.348622:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204409.348627:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5864 00000100:00080000:0.0:1788204409.348631:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204409.348644:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204409.348657:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204409.348671:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187952768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204409.348673:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187952896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204409.348687:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204409.348690:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204409.348693:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204409.348697:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5864 00000100:00080000:0.0:1788204409.348701:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204409.348706:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204409.348712:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204409.348724:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187953024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204409.348735:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204409.348738:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204409.348741:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204409.348745:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5864 00000100:00080000:0.0:1788204409.348749:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204409.348755:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:2.0:1788204409.348755:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788204409.348762:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204409.348782:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187953152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204409.348793:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204409.348796:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204409.348798:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204409.348801:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5864 00000100:00080000:0.0:1788204409.348805:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788204409.348807:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204409/real 1788204409] req@ffff8e828793d880 x1875062187952768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204409.348812:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204409.348821:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204409.348829:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187952768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204409.348837:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187953280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204409.348838:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204409.348847:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204409.348854:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187952768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204409.348882:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187952896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204409.348900:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204409.348908:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204409/real 1788204409] req@ffff8e828793fb80 x1875062187952896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204409.348916:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187952896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204409.348922:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204409.348924:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204409.348926:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187952896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204409.348929:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187953024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204409.348940:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204409.348945:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204409/real 1788204409] req@ffff8e828793d500 x1875062187953024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204409.348952:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187953024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204409.348956:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204409.348958:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204409.348959:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187953024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204409.348963:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187953152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204409.348973:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204409.348978:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204409/real 1788204409] req@ffff8e828793e300 x1875062187953152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204409.348985:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187953152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204409.348989:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204409.348991:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204409.348993:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187953152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204409.348996:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187953280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204409.349004:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204409.349009:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204409/real 1788204409] req@ffff8e828793c380 x1875062187953280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204409.349015:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187953280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204409.349020:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204409.349022:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204409.349023:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187953280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204410.052476:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204411.076532:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204412.100553:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204413.124549:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204414.148501:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204414.468412:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204414.468424:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204414.468429:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204414.468441:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5869 00000100:00080000:2.0:1788204414.468454:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204414.468462:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204414.468505:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187953408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204414.468532:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204414.468536:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204414.468538:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204414.468543:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5869 00000100:00080000:2.0:1788204414.468547:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204414.468553:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204414.468564:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204414.468571:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187953408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204414.468577:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187953536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204414.468588:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204414.468592:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204414.468593:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204414.468595:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5869 00000100:00080000:2.0:1788204414.468598:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204414.468600:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204414.468603:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204414.468612:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187953664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204414.468618:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204414.468620:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204414.468621:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204414.468623:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5869 00000100:00080000:2.0:1788204414.468626:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204414.468633:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204414.468637:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204414.468643:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187953792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204414.468649:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204414.468651:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204414.468652:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204414.468654:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5869 00000100:00080000:2.0:1788204414.468656:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204414.468658:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204414.468660:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204414.468669:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187953920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204414.468669:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204414.468730:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204414/real 1788204414] req@ffff8e82894e7800 x1875062187953408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204414.468748:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187953408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204414.468758:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204414.468763:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204414.468767:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187953408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204414.468793:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187953536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204414.468816:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204414.468828:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204414/real 1788204414] req@ffff8e82894e6300 x1875062187953536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204414.468848:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187953536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204414.468859:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204414.468862:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204414.468865:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187953536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204414.468870:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187953664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204414.468892:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204414.468902:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204414/real 1788204414] req@ffff8e82894e4a80 x1875062187953664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204414.468918:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187953664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204414.468926:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204414.468929:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204414.468932:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187953664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204414.468937:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187953792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204414.468956:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204414.468967:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204414/real 1788204414] req@ffff8e82894e4e00 x1875062187953792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204414.468977:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187953792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204414.468984:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204414.468987:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204414.468990:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187953792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204414.468996:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187953920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204414.469014:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204414.469024:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204414/real 1788204414] req@ffff8e82894e6d80 x1875062187953920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204414.469035:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187953920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204414.469049:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204414.469052:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204414.469055:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187953920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204415.172458:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204416.196520:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204417.220484:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204418.244525:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204419.268494:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204419.588441:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204419.588454:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204419.588459:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204419.588472:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5874 00000100:00080000:0.0:1788204419.588485:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204419.588493:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204419.588535:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187954048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204419.588563:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204419.588566:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204419.588568:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204419.588572:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5874 00000100:00080000:0.0:1788204419.588576:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204419.588580:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204419.588591:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204419.588595:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187954048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204419.588605:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187954176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204419.588615:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204419.588618:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204419.588619:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204419.588622:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5874 00000100:00080000:0.0:1788204419.588626:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204419.588630:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204419.588635:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204419.588648:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187954304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204419.588663:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204419.588667:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204419.588670:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788204419.588671:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204419.588675:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5874 00000100:00080000:0.0:1788204419.588679:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204419.588682:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204419.588686:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204419.588695:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187954432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204419.588701:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204419.588703:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204419.588704:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204419.588706:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5874 00000100:00080000:0.0:1788204419.588708:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204419.588710:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204419.588713:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204419.588719:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187954560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204419.588730:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204419/real 1788204419] req@ffff8e828793f480 x1875062187954048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204419.588753:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187954048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204419.588765:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204419.588768:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204419.588771:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187954048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204419.588796:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187954176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204419.588820:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204419.588833:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204419/real 1788204419] req@ffff8e828793dc00 x1875062187954176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204419.588843:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187954176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204419.588848:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204419.588851:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204419.588853:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187954176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204419.588856:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187954304:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204419.588870:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204419.588877:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204419/real 1788204419] req@ffff8e828793c000 x1875062187954304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204419.588883:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187954304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204419.588888:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204419.588889:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204419.588891:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187954304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204419.588895:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187954432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204419.588906:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204419.588915:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204419/real 1788204419] req@ffff8e828793ca80 x1875062187954432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204419.588926:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187954432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204419.588933:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204419.588936:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204419.588940:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187954432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204419.588946:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187954560:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204419.588963:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204419.588970:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204419/real 1788204419] req@ffff8e828793f800 x1875062187954560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204419.588977:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187954560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204419.588983:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204419.588986:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204419.588988:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187954560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204420.292513:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204421.316525:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204422.340527:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204423.364494:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204424.388496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204424.708434:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204424.708448:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204424.708454:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204424.708468:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5879 00000100:00080000:0.0:1788204424.708482:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204424.708491:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204424.708549:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187954688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204424.708587:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204424.708592:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204424.708594:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204424.708599:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5879 00000100:00080000:0.0:1788204424.708604:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204424.708609:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204424.708621:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204424.708635:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187954688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204424.708639:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187954816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204424.708652:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204424.708656:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204424.708658:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204424.708662:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5879 00000100:00080000:0.0:1788204424.708665:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204424.708671:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204424.708676:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204424.708688:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187954944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204424.708704:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204424.708708:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204424.708710:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204424.708714:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5879 00000100:00080000:0.0:1788204424.708718:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788204424.708725:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204424.708726:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204424.708733:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204424.708744:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187955072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204424.708754:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204424.708758:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204424.708760:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204424.708763:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5879 00000100:00080000:0.0:1788204424.708767:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204424.708778:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204424.708783:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204424.708786:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204424/real 1788204424] req@ffff8e828793e680 x1875062187954688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204424.708807:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187955200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204424.708811:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187954688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204424.708839:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204424.708843:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204424.708847:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187954688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204424.708873:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187954816:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204424.708897:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204424.708910:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204424/real 1788204424] req@ffff8e828793c700 x1875062187954816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204424.708922:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187954816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204424.708930:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204424.708934:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204424.708937:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187954816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204424.708943:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187954944:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204424.708961:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204424.708972:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204424/real 1788204424] req@ffff8e828793f100 x1875062187954944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204424.708983:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187954944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204424.708991:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204424.708995:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204424.708998:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187954944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204424.709005:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187955072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204424.709022:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204424.709032:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204424/real 1788204424] req@ffff8e828793ce00 x1875062187955072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204424.709043:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187955072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204424.709051:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204424.709054:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204424.709058:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187955072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204424.709064:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187955200:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204424.709096:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204424.709107:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204424/real 1788204424] req@ffff8e828793f800 x1875062187955200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204424.709116:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187955200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204424.709121:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204424.709123:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204424.709125:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187955200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204425.412497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204426.436516:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204427.460458:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204428.484505:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204429.508453:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204429.828452:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204429.828462:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204429.828468:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204429.828478:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5884 00000100:00080000:0.0:1788204429.828490:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204429.828496:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204429.828536:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187955328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204429.828556:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204429.828559:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204429.828560:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204429.828563:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5884 00000100:00080000:0.0:1788204429.828565:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204429.828568:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204429.828584:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204429.828587:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187955328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204429.828600:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187955456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204429.828612:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204429.828616:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204429.828618:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204429.828621:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5884 00000100:00080000:0.0:1788204429.828625:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204429.828628:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204429.828632:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204429.828641:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187955584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204429.828652:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204429.828654:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204429.828655:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204429.828658:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5884 00000100:00080000:0.0:1788204429.828660:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788204429.828663:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204429.828666:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204429.828671:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204429.828680:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187955712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204429.828685:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204429.828688:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204429.828689:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204429.828692:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5884 00000100:00080000:0.0:1788204429.828694:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204429.828697:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204429.828699:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204429.828706:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187955840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204429.828712:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204429/real 1788204429] req@ffff8e828793ca80 x1875062187955328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204429.828728:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187955328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204429.828736:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204429.828739:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204429.828743:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187955328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204429.828782:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187955456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204429.828806:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204429.828817:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204429/real 1788204429] req@ffff8e828793c000 x1875062187955456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204429.828828:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187955456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204429.828836:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204429.828838:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204429.828841:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187955456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204429.828847:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187955584:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204429.828867:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204429.828878:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204429/real 1788204429] req@ffff8e828793dc00 x1875062187955584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204429.828889:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187955584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204429.828898:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204429.828900:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204429.828902:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187955584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204429.828906:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187955712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204429.828920:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204429.828926:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204429/real 1788204429] req@ffff8e828793f480 x1875062187955712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204429.828932:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187955712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204429.828937:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204429.828939:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204429.828941:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187955712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204429.828944:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187955840:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204429.828953:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204429.828958:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204429/real 1788204429] req@ffff8e828793c380 x1875062187955840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204429.828964:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187955840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204429.828969:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204429.828970:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204429.828972:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187955840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204430.532487:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204431.556450:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204432.580467:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204433.604503:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204434.628444:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204434.948443:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204434.948457:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204434.948463:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204434.948474:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5890 00000100:00080000:0.0:1788204434.948486:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204434.948493:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204434.948532:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187955968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204434.948561:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204434.948566:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204434.948568:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204434.948573:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5890 00000100:00080000:0.0:1788204434.948577:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204434.948582:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204434.948595:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204434.948612:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187956096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204434.948618:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187955968:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204434.948629:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204434.948635:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204434.948637:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204434.948641:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5890 00000100:00080000:0.0:1788204434.948644:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204434.948648:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204434.948653:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204434.948665:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187956224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204434.948681:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204434.948685:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204434.948688:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204434.948693:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5890 00000100:00080000:0.0:1788204434.948698:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204434.948703:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204434.948710:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204434.948721:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187956352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204434.948727:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204434.948730:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204434.948732:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204434.948736:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5890 00000100:00080000:0.0:1788204434.948740:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204434.948743:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204434.948748:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204434.948764:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187956096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204434.948767:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187956480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204434.948805:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187956224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204434.948830:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187956352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204434.948860:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187956480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204435.652499:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204436.676447:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204437.700541:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788204438.020632:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:1.0:1788204438.020676:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788204438.020692:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788204438.020745:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788204438.020755:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788204438.020763:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788204438.020770:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:2.0:1788204438.020899:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204434/real 1788204438] req@ffff8e828793c380 x1875062187956480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204438.020925:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187956480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204438.020944:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204438.020950:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204438.020954:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187956480:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204438.020962:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204434/real 1788204438] req@ffff8e828793d500 x1875062187956096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204438.020976:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187956096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204438.020984:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204438.020988:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204438.020991:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187956096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204438.020998:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204434/real 1788204438] req@ffff8e828793fb80 x1875062187956224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204438.021010:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187956224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204438.021018:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204438.021021:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204438.021025:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187956224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204438.021031:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204434/real 1788204438] req@ffff8e828793d880 x1875062187956352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204438.021044:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187956352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204438.021051:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204438.021054:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204438.021058:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187956352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204438.021063:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204434/real 1788204438] req@ffff8e828793e300 x1875062187955968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204438.021075:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187955968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204438.021083:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204438.021086:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204438.021089:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187955968:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788204438.724493:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204439.748485:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204440.068452:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204440.068461:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204440.068465:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204440.068475:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5895 00000100:00080000:1.0:1788204440.068485:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204440.068492:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204440.068523:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187956608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204440.068552:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204440.068556:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204440.068558:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204440.068562:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5895 00000100:00080000:1.0:1788204440.068565:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204440.068569:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204440.068576:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204440.068588:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187956736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204440.068596:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204440.068599:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204440.068601:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204440.068604:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5895 00000100:00080000:1.0:1788204440.068607:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204440.068610:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204440.068614:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204440.068624:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187956864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204440.068633:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187956608:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204440.068635:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204440.068642:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204440.068643:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204440.068647:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5895 00000100:00080000:1.0:1788204440.068651:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204440.068655:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204440.068660:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204440.068672:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187956992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204440.068680:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204440.068683:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204440.068684:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204440.068688:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5895 00000100:00080000:1.0:1788204440.068694:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204440.068698:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204440.068703:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788204440.068707:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204440.068728:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187957120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204440.068757:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204440/real 1788204440] req@ffff8e828d722300 x1875062187956608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204440.068775:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187956608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204440.068786:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204440.068790:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204440.068793:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187956608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204440.068800:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187956736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204440.068850:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204440.068862:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204440/real 1788204440] req@ffff8e828d721180 x1875062187956736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204440.068870:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187956736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204440.068876:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204440.068878:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204440.068880:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187956736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204440.068909:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187956864:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204440.068928:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204440.068938:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204440/real 1788204440] req@ffff8e828d722d80 x1875062187956864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204440.068949:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187956864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204440.068956:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204440.068958:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204440.068961:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187956864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204440.068967:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187956992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204440.068985:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204440.068996:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204440/real 1788204440] req@ffff8e828d720380 x1875062187956992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204440.069008:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187956992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204440.069015:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204440.069018:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204440.069021:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187956992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204440.069026:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187957120:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204440.069044:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204440.069055:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204440/real 1788204440] req@ffff8e828d720a80 x1875062187957120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204440.069065:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187957120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204440.069079:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204440.069082:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204440.069085:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187957120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204440.772508:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204441.796465:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204442.820552:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204443.844509:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204444.868569:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204445.188449:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204445.188460:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204445.188465:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204445.188476:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5900 00000100:00080000:1.0:1788204445.188486:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204445.188495:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204445.188530:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187957248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204445.188559:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204445.188563:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204445.188565:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204445.188569:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5900 00000100:00080000:1.0:1788204445.188580:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204445.188586:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204445.188596:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204445.188610:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187957248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204445.188611:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187957376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204445.188633:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204445.188637:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204445.188639:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204445.188643:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5900 00000100:00080000:1.0:1788204445.188647:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204445.188651:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204445.188656:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204445.188669:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187957504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204445.188679:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788204445.188681:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788204445.188686:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204445.188688:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204445.188693:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5900 00000100:00080000:1.0:1788204445.188698:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204445.188702:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204445.188708:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204445.188719:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187957632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204445.188745:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788204445.188749:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204445/real 1788204445] req@ffff8e828d723800 x1875062187957248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204445.188750:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204445.188754:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204445.188763:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5900 00000100:00080000:1.0:1788204445.188769:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788204445.188773:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187957248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204445.188777:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:2.0:1788204445.188787:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1788204445.188788:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788204445.188792:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204445.188795:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187957248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204445.188804:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187957760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204445.188816:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187957376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204445.188835:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204445.188843:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204445/real 1788204445] req@ffff8e828d721500 x1875062187957376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204445.188851:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187957376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204445.188856:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204445.188859:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204445.188860:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187957376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204445.188864:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187957504:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204445.188874:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204445.188880:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204445/real 1788204445] req@ffff8e828d720000 x1875062187957504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204445.188886:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187957504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204445.188891:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204445.188892:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204445.188894:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187957504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204445.188897:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187957632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204445.188906:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204445.188911:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204445/real 1788204445] req@ffff8e828d722680 x1875062187957632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204445.188917:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187957632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204445.188922:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204445.188923:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204445.188925:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187957632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204445.188935:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187957760:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204445.188944:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204445.188949:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204445/real 1788204445] req@ffff8e828d720e00 x1875062187957760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204445.188956:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187957760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204445.188960:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204445.188961:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204445.188963:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187957760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204445.892580:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204446.916507:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204447.940514:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204448.964560:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204449.988508:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204450.308453:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204450.308463:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204450.308468:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204450.308478:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5905 00000100:00080000:3.0:1788204450.308489:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204450.308496:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204450.308535:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187957888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204450.308556:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204450.308559:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204450.308560:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204450.308563:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5905 00000100:00080000:3.0:1788204450.308565:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204450.308567:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204450.308576:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204450.308583:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187958016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204450.308590:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204450.308592:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204450.308594:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204450.308597:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5905 00000100:00080000:3.0:1788204450.308599:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204450.308601:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204450.308604:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204450.308611:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187958144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204450.308617:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204450.308619:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204450.308620:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204450.308622:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5905 00000100:00080000:3.0:1788204450.308624:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204450.308627:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204450.308631:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204450.308661:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187958272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204450.308664:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187957888:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204450.308684:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204450.308687:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204450.308689:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204450.308693:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5905 00000100:00080000:3.0:1788204450.308695:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204450.308699:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204450.308703:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204450.308712:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187958400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204450.308747:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204450.308791:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204450/real 1788204450] req@ffff8e8289538a80 x1875062187957888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204450.308807:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187957888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204450.308817:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204450.308821:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204450.308825:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187957888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204450.308832:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187958016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204450.308852:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204450.308863:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204450/real 1788204450] req@ffff8e828953bb80 x1875062187958016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204450.308874:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187958016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204450.308887:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204450.308891:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204450.308894:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187958016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204450.308899:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187958144:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204450.308917:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204450.308928:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204450/real 1788204450] req@ffff8e8289539880 x1875062187958144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204450.308939:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187958144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204450.308945:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204450.308949:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204450.308952:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187958144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204450.308975:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187958272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204450.308993:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204450.309003:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204450/real 1788204450] req@ffff8e828953b100 x1875062187958272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204450.309014:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187958272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204450.309021:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204450.309023:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204450.309027:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187958272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204450.309033:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187958400:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204450.309050:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204450.309060:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204450/real 1788204450] req@ffff8e8289539500 x1875062187958400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204450.309070:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187958400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204450.309077:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204450.309079:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204450.309082:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187958400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204451.012534:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204452.036497:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204453.060538:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204454.084506:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204455.108536:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204455.428446:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204455.428456:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204455.428460:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204455.428469:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5910 00000100:00080000:3.0:1788204455.428482:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204455.428490:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204455.428534:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187958528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204455.428574:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204455.428578:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204455.428580:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204455.428585:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5910 00000100:00080000:3.0:1788204455.428589:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204455.428593:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204455.428602:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204455.428615:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187958656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204455.428622:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204455.428625:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204455.428626:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204455.428629:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5910 00000100:00080000:3.0:1788204455.428631:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204455.428634:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204455.428637:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204455.428644:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187958784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204455.428652:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204455.428655:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204455.428659:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788204455.428663:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187958528:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204455.428663:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5910 00000100:00080000:3.0:1788204455.428666:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204455.428672:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204455.428680:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204455.428689:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187958912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204455.428699:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204455.428703:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204455.428705:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204455.428710:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5910 00000100:00080000:3.0:1788204455.428727:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204455.428732:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:2.0:1788204455.428737:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:3.0:1788204455.428737:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204455.428754:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187959040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204455.428790:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204455/real 1788204455] req@ffff8e8289539f80 x1875062187958528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204455.428804:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187958528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204455.428813:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204455.428817:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204455.428821:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187958528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204455.428828:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187958656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204455.428844:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204455.428852:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204455/real 1788204455] req@ffff8e828953a680 x1875062187958656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204455.428859:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187958656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204455.428864:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204455.428866:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204455.428868:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187958656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204455.428885:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187958784:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204455.428895:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204455.428901:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204455/real 1788204455] req@ffff8e8289538380 x1875062187958784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204455.428913:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187958784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204455.428918:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204455.428921:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204455.428922:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187958784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204455.428926:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187958912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204455.428937:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204455.428943:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204455/real 1788204455] req@ffff8e8289539500 x1875062187958912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204455.428949:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187958912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204455.428954:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204455.428956:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204455.428957:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187958912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204455.428960:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187959040:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204455.428969:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204455.428974:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204455/real 1788204455] req@ffff8e828953b100 x1875062187959040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204455.428980:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187959040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204455.428990:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204455.428992:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204455.428994:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187959040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204456.132449:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204457.156490:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204458.180563:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204459.204520:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204460.228499:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204460.548492:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204460.548506:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204460.548512:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204460.548522:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5915 00000100:00080000:0.0:1788204460.548534:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204460.548541:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204460.548590:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187959168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204460.548616:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204460.548619:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204460.548621:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204460.548624:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5915 00000100:00080000:0.0:1788204460.548627:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204460.548629:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204460.548637:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204460.548646:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187959296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204460.548654:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204460.548656:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204460.548658:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204460.548660:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5915 00000100:00080000:0.0:1788204460.548662:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204460.548665:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204460.548669:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204460.548675:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187959168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204460.548679:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187959424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204460.548686:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204460.548688:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204460.548689:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204460.548691:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5915 00000100:00080000:0.0:1788204460.548693:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204460.548695:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204460.548698:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204460.548705:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187959552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204460.548710:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204460.548712:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204460.548713:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204460.548716:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5915 00000100:00080000:0.0:1788204460.548717:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204460.548725:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204460.548728:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204460.548737:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187959680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204460.548748:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204460.548808:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204460/real 1788204460] req@ffff8e828793f480 x1875062187959168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204460.548823:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187959168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204460.548832:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204460.548836:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204460.548839:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187959168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204460.548847:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187959296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204460.548871:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204460.548885:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204460/real 1788204460] req@ffff8e828793dc00 x1875062187959296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204460.548897:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187959296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204460.548906:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204460.548908:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204460.548910:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187959296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204460.548933:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187959424:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204460.548947:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204460.548954:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204460/real 1788204460] req@ffff8e828793c000 x1875062187959424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204460.548961:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187959424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204460.548971:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204460.548973:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204460.548975:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187959424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204460.548979:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187959552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204460.548990:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204460.548996:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204460/real 1788204460] req@ffff8e828793ca80 x1875062187959552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204460.549004:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187959552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204460.549008:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204460.549010:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204460.549011:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187959552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204460.549014:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187959680:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204460.549024:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204460.549029:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204460/real 1788204460] req@ffff8e828793f800 x1875062187959680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204460.549035:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187959680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204460.549040:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204460.549041:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204460.549043:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187959680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204461.252497:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204462.276505:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204463.300517:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204464.324551:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204465.348562:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204465.668417:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204465.668430:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204465.668435:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204465.668448:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5920 00000100:00080000:3.0:1788204465.668461:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204465.668469:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204465.668520:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187959808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204465.668548:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204465.668552:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204465.668554:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204465.668558:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5920 00000100:00080000:3.0:1788204465.668561:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204465.668564:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204465.668574:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204465.668587:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187959936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204465.668590:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187959808:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204465.668598:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204465.668601:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204465.668602:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204465.668605:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5920 00000100:00080000:3.0:1788204465.668607:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204465.668609:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204465.668612:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204465.668620:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187960064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204465.668626:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204465.668629:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204465.668630:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204465.668632:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5920 00000100:00080000:3.0:1788204465.668634:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204465.668636:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204465.668639:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204465.668645:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187960192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204465.668650:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204465.668651:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204465.668652:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204465.668655:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5920 00000100:00080000:3.0:1788204465.668658:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204465.668660:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204465.668662:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204465.668668:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187960320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204465.668671:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204465.668720:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204465/real 1788204465] req@ffff8e8289539880 x1875062187959808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204465.668735:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187959808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204465.668747:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204465.668750:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204465.668754:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187959808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204465.668773:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187959936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204465.668788:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204465.668796:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204465/real 1788204465] req@ffff8e828953bb80 x1875062187959936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204465.668803:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187959936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204465.668808:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204465.668810:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204465.668812:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187959936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204465.668815:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187960064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204465.668829:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204465.668835:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204465/real 1788204465] req@ffff8e8289538a80 x1875062187960064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204465.668842:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187960064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204465.668846:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204465.668848:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204465.668850:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187960064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204465.668855:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187960192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204465.668873:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204465.668883:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204465/real 1788204465] req@ffff8e828953b100 x1875062187960192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204465.668895:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187960192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204465.668902:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204465.668905:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204465.668907:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187960192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204465.668910:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187960320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204465.668924:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204465.668930:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204465/real 1788204465] req@ffff8e8289539500 x1875062187960320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204465.668936:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187960320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204465.668941:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204465.668943:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204465.668944:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187960320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204466.372443:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204467.396504:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204468.420513:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204469.444515:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204470.468525:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204470.788472:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204470.788482:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204470.788487:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204470.788502:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5925 00000100:00080000:3.0:1788204470.788517:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204470.788525:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204470.788563:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187960448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204470.788599:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204470.788604:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204470.788606:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204470.788611:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5925 00000100:00080000:3.0:1788204470.788616:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204470.788621:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204470.788632:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204470.788640:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187960448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204470.788652:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187960576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204470.788666:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204470.788670:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204470.788672:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204470.788676:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5925 00000100:00080000:3.0:1788204470.788681:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204470.788684:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204470.788688:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204470.788700:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187960704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204470.788709:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204470.788712:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204470.788714:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788204470.788719:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788204470.788720:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5925 00000100:00080000:3.0:1788204470.788727:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788204470.788759:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204470/real 1788204470] req@ffff8e8289538380 x1875062187960448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204470.788774:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187960448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204470.788783:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204470.788786:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204470.788790:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187960448:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204470.788798:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204470.788806:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204470.788813:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187960576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204470.788822:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187960832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204470.788833:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788204470.788836:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788204470.788846:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204470/real 1788204470] req@ffff8e828953a680 x1875062187960576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204470.788848:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204470.788851:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204470.788859:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5925 00000100:00080000:3.0:1788204470.788866:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788204470.788872:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187960576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204470.788872:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204470.788886:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788204470.788887:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204470.788895:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204470.788897:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187960576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204470.788902:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187960704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204470.788918:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204470.788926:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204470/real 1788204470] req@ffff8e8289539f80 x1875062187960704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204470.788938:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187960704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204470.788943:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204470.788945:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204470.788947:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187960704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204470.788956:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187960832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204470.788969:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204470.788976:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204470/real 1788204470] req@ffff8e8289538380 x1875062187960832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204470.788984:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187960832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204470.788990:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204470.788993:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204470.788997:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187960832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204470.789009:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187960960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204470.789043:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187960960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204470.789069:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204470.789083:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204470/real 1788204470] req@ffff8e8289539500 x1875062187960960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204470.789094:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187960960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204470.789109:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204470.789113:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204470.789117:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187960960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204471.492487:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204472.516450:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204473.540451:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204474.564507:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204475.588512:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204475.908445:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204475.908459:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204475.908465:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204475.908477:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5930 00000100:00080000:1.0:1788204475.908491:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204475.908500:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204475.908552:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187961088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204475.908581:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204475.908586:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204475.908587:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204475.908592:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5930 00000100:00080000:1.0:1788204475.908596:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204475.908601:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204475.908613:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204475.908629:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187961216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204475.908633:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187961088:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204475.908652:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204475.908656:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204475.908658:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204475.908663:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5930 00000100:00080000:1.0:1788204475.908666:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204475.908671:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204475.908677:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204475.908690:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187961344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204475.908699:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204475.908703:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204475.908705:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204475.908709:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5930 00000100:00080000:1.0:1788204475.908713:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788204475.908717:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788204475.908721:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204475.908728:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204475.908740:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187961472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204475.908749:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204475.908753:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204475.908754:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204475.908760:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5930 00000100:00080000:1.0:1788204475.908763:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788204475.908765:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204475/real 1788204475] req@ffff8e828d720e00 x1875062187961088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204475.908770:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204475.908779:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204475.908785:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187961088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204475.908794:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187961600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204475.908795:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204475.908803:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204475.908806:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187961088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204475.908841:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187961216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204475.908857:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204475.908865:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204475/real 1788204475] req@ffff8e828d722680 x1875062187961216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204475.908876:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187961216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204475.908881:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204475.908883:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204475.908886:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187961216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204475.908890:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187961344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204475.908901:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204475.908906:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204475/real 1788204475] req@ffff8e828d720000 x1875062187961344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204475.908913:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187961344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204475.908918:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204475.908919:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204475.908921:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187961344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204475.908924:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187961472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204475.908934:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204475.908939:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204475/real 1788204475] req@ffff8e828d721500 x1875062187961472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204475.908945:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187961472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204475.908949:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204475.908951:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204475.908952:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187961472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204475.908955:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187961600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204475.908963:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204475.908968:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204475/real 1788204475] req@ffff8e828d723800 x1875062187961600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204475.908975:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187961600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204475.908979:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204475.908981:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204475.908982:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187961600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204476.612500:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204477.636524:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204478.660510:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204479.684473:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204480.708481:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204481.028415:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204481.028428:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204481.028433:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204481.028446:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5936 00000100:00080000:3.0:1788204481.028459:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204481.028467:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204481.028511:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187961728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204481.028549:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204481.028553:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204481.028555:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204481.028560:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5936 00000100:00080000:3.0:1788204481.028565:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204481.028570:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204481.028581:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204481.028586:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187961728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204481.028601:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187961856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204481.028613:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204481.028616:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204481.028618:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204481.028622:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5936 00000100:00080000:3.0:1788204481.028625:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204481.028630:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204481.028635:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204481.028647:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187961984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204481.028656:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788204481.028659:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788204481.028662:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204481.028666:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204481.028671:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5936 00000100:00080000:3.0:1788204481.028676:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204481.028681:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204481.028686:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204481.028698:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187962112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204481.028707:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204481.028711:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204481.028713:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788204481.028716:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204481/real 1788204481] req@ffff8e828953b100 x1875062187961728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204481.028717:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5936 00000100:00080000:3.0:1788204481.028723:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204481.028727:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204481.028733:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204481.028749:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187962240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204481.028752:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187961728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204481.028780:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204481.028784:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204481.028788:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187961728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204481.028813:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187961856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204481.028832:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204481.028843:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204481/real 1788204481] req@ffff8e8289538a80 x1875062187961856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204481.028855:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187961856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204481.028870:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204481.028873:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204481.028877:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187961856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204481.028883:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187961984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204481.028902:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204481.028911:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204481/real 1788204481] req@ffff8e828953bb80 x1875062187961984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204481.028919:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187961984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204481.028925:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204481.028927:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204481.028929:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187961984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204481.028939:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187962112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204481.028952:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204481.028958:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204481/real 1788204481] req@ffff8e8289539880 x1875062187962112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204481.028965:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187962112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204481.028969:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204481.028971:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204481.028973:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187962112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204481.028977:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187962240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204481.028987:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204481.028992:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204481/real 1788204481] req@ffff8e8289539500 x1875062187962240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204481.028999:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187962240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204481.029004:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204481.029006:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204481.029008:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187962240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204481.732507:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204482.756519:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204483.780544:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204484.804539:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204485.828505:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204486.148438:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204486.148449:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204486.148453:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204486.148464:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5941 00000100:00080000:0.0:1788204486.148476:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204486.148483:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204486.148525:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187962368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204486.148557:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204486.148560:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204486.148561:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204486.148565:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5941 00000100:00080000:0.0:1788204486.148568:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204486.148571:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204486.148580:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204486.148589:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187962496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204486.148603:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204486.148606:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788204486.148606:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187962368:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204486.148610:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204486.148616:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5941 00000100:00080000:0.0:1788204486.148618:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204486.148620:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204486.148623:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204486.148631:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187962624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204486.148637:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204486.148643:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204486.148645:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204486.148647:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5941 00000100:00080000:0.0:1788204486.148649:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204486.148652:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204486.148656:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204486.148662:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187962752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204486.148668:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204486.148671:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204486.148672:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204486.148675:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5941 00000100:00080000:0.0:1788204486.148678:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788204486.148681:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204486.148683:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204486.148687:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204486.148695:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187962880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204486.148729:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204486/real 1788204486] req@ffff8e828793ce00 x1875062187962368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204486.148744:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187962368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204486.148753:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204486.148757:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204486.148761:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187962368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204486.148786:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187962496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204486.148809:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204486.148818:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204486/real 1788204486] req@ffff8e828793f100 x1875062187962496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204486.148826:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187962496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204486.148832:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204486.148834:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204486.148836:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187962496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204486.148840:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187962624:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204486.148854:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204486.148864:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204486/real 1788204486] req@ffff8e828793c700 x1875062187962624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204486.148876:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187962624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204486.148889:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204486.148893:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204486.148896:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187962624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204486.148902:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187962752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204486.148923:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204486.148934:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204486/real 1788204486] req@ffff8e828793e680 x1875062187962752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204486.148946:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187962752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204486.148954:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204486.148957:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204486.148961:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187962752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204486.148966:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187962880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204486.148987:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204486.148998:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204486/real 1788204486] req@ffff8e828793f800 x1875062187962880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204486.149009:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187962880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204486.149044:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204486.149048:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204486.149052:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187962880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204486.852500:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204487.876520:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204488.900517:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204489.924523:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204490.948480:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204491.268419:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204491.268428:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204491.268433:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204491.268444:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5946 00000100:00080000:3.0:1788204491.268455:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204491.268462:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204491.268505:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187963008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204491.268527:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204491.268532:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204491.268534:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204491.268537:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5946 00000100:00080000:3.0:1788204491.268539:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204491.268542:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204491.268551:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204491.268561:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187963136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204491.268568:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788204491.268574:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187963008:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204491.268580:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204491.268586:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204491.268590:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5946 00000100:00080000:3.0:1788204491.268595:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204491.268599:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204491.268604:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204491.268618:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187963264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204491.268628:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204491.268631:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204491.268633:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204491.268637:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5946 00000800:00000100:2.0:1788204491.268642:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788204491.268643:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204491.268650:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204491.268657:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204491.268668:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187963392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204491.268675:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204491.268677:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204491.268679:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204491.268682:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5946 00000100:00080000:3.0:1788204491.268685:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788204491.268686:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204491/real 1788204491] req@ffff8e8289538380 x1875062187963008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204491.268690:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204491.268697:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204491.268706:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187963520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204491.268710:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187963008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204491.268725:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204491.268730:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204491.268735:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187963008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204491.268766:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187963136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204491.268785:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204491.268794:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204491/real 1788204491] req@ffff8e8289539f80 x1875062187963136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204491.268802:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187963136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204491.268808:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204491.268811:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204491.268815:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187963136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204491.268820:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187963264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204491.268839:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204491.268850:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204491/real 1788204491] req@ffff8e828953a680 x1875062187963264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204491.268859:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187963264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204491.268865:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204491.268869:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204491.268872:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187963264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204491.268877:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187963392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204491.268897:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204491.268908:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204491/real 1788204491] req@ffff8e8289539500 x1875062187963392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204491.268918:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187963392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204491.268923:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204491.268925:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204491.268940:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187963392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204491.268944:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187963520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204491.268957:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204491.268964:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204491/real 1788204491] req@ffff8e8289539880 x1875062187963520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204491.268971:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187963520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204491.268976:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204491.268978:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204491.268980:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187963520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204491.972502:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204492.996499:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204494.020512:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204495.044482:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204496.068525:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204496.388420:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204496.388438:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204496.388443:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204496.388453:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5951 00000100:00080000:3.0:1788204496.388465:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204496.388472:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204496.388514:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187963648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204496.388544:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204496.388549:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204496.388551:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204496.388555:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5951 00000100:00080000:3.0:1788204496.388559:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204496.388564:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204496.388584:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204496.388605:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187963776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204496.388614:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187963648:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204496.388617:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204496.388623:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204496.388625:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204496.388628:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5951 00000100:00080000:3.0:1788204496.388631:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204496.388633:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204496.388637:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204496.388647:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187963904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204496.388653:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204496.388655:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204496.388656:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204496.388658:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5951 00000100:00080000:3.0:1788204496.388660:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204496.388663:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204496.388666:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204496.388672:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187964032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204496.388681:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204496.388683:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204496.388684:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204496.388686:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5951 00000100:00080000:3.0:1788204496.388689:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204496.388691:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204496.388694:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204496.388700:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187964160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204496.388707:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204496.388781:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204496/real 1788204496] req@ffff8e828953bb80 x1875062187963648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204496.388801:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187963648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204496.388812:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204496.388817:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204496.388821:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187963648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204496.388871:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187963776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204496.388894:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204496.388906:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204496/real 1788204496] req@ffff8e8289538a80 x1875062187963776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204496.388918:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187963776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204496.388925:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204496.388928:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204496.388931:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187963776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204496.388937:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187963904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204496.388952:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204496.388960:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204496/real 1788204496] req@ffff8e828953b100 x1875062187963904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204496.388970:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187963904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204496.388976:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204496.388979:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204496.388981:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187963904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204496.388986:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187964032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204496.389009:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204496.389018:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204496/real 1788204496] req@ffff8e8289539880 x1875062187964032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204496.389028:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187964032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204496.389034:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204496.389037:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204496.389040:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187964032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204496.389045:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187964160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204496.389059:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204496.389067:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204496/real 1788204496] req@ffff8e8289539500 x1875062187964160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204496.389078:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187964160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204496.389085:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204496.389087:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204496.389090:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187964160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204497.092496:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204498.116524:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204499.140508:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204500.164545:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204501.188561:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204501.508445:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204501.508456:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204501.508460:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204501.508470:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5956 00000100:00080000:1.0:1788204501.508481:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204501.508487:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204501.508523:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187964288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204501.508551:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204501.508555:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204501.508557:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204501.508561:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5956 00000100:00080000:1.0:1788204501.508565:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204501.508569:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204501.508578:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204501.508590:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187964416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204501.508595:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187964288:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204501.508605:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204501.508609:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204501.508611:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204501.508615:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5956 00000100:00080000:1.0:1788204501.508619:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204501.508623:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204501.508629:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204501.508640:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187964544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204501.508648:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204501.508652:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204501.508653:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204501.508657:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5956 00000100:00080000:1.0:1788204501.508660:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204501.508664:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204501.508669:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204501.508681:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187964672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204501.508690:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204501.508694:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788204501.508695:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187964416:192.168.201.121@tcp:38:kworker.0 00000100:00080000:1.0:1788204501.508697:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204501.508703:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5956 00000100:00080000:1.0:1788204501.508706:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204501.508710:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204501.508716:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204501.508731:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187964544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204501.508732:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187964800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204501.508774:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187964672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204501.508797:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187964800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204502.212455:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204503.236461:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204504.260528:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788204504.580664:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:1.0:1788204504.580721:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788204504.580737:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788204504.580776:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788204504.580785:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788204504.580792:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788204504.580802:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:2.0:1788204504.580829:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204501/real 1788204504] req@ffff8e828d722300 x1875062187964800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204504.580867:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187964800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204504.580879:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204504.580885:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204504.580889:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187964800:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204504.580897:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204501/real 1788204504] req@ffff8e828d720380 x1875062187964416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204504.580915:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187964416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204504.580923:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204504.580926:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204504.580929:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187964416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204504.580935:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204501/real 1788204504] req@ffff8e828d722d80 x1875062187964544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204504.580954:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187964544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204504.580962:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204504.580966:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204504.580968:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187964544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204504.580974:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204501/real 1788204504] req@ffff8e828d721180 x1875062187964672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204504.580984:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187964672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204504.580991:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204504.580994:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204504.580998:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187964672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204504.581004:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204501/real 1788204504] req@ffff8e828d720a80 x1875062187964288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204504.581016:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187964288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204504.581022:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204504.581026:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204504.581029:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187964288:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788204505.284506:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204506.308546:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204506.628439:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204506.628449:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204506.628453:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204506.628463:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5961 00000100:00080000:2.0:1788204506.628474:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204506.628480:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204506.628509:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187964928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204506.628538:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204506.628542:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204506.628543:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204506.628548:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5961 00000100:00080000:2.0:1788204506.628552:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204506.628557:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204506.628566:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204506.628580:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187964928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204506.628585:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187965056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204506.628602:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204506.628606:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204506.628607:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204506.628611:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5961 00000100:00080000:2.0:1788204506.628614:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204506.628618:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204506.628622:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204506.628632:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187965184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204506.628641:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204506.628643:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204506.628645:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204506.628649:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5961 00000100:00080000:2.0:1788204506.628653:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788204506.628653:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204506.628671:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204506.628679:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204506.628696:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187965312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204506.628705:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204506.628708:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204506.628710:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204506.628714:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5961 00000100:00080000:2.0:1788204506.628720:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204506.628723:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204506/real 1788204506] req@ffff8e82894e7100 x1875062187964928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204506.628728:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204506.628736:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204506.628752:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187965440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204506.628757:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187964928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204506.628773:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204506.628778:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204506.628783:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187964928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204506.628812:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187965056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204506.628846:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204506.628862:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204506/real 1788204506] req@ffff8e82894e5c00 x1875062187965056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204506.628877:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187965056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204506.628888:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204506.628892:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204506.628895:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187965056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204506.628901:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187965184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204506.628923:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204506.628933:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204506/real 1788204506] req@ffff8e82894e4380 x1875062187965184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204506.628945:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187965184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204506.628953:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204506.628956:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204506.628959:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187965184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204506.628965:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187965312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204506.628985:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204506.628996:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204506/real 1788204506] req@ffff8e82894e5f80 x1875062187965312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204506.629007:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187965312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204506.629021:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204506.629024:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204506.629027:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187965312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204506.629052:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187965440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204506.629069:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204506.629080:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204506/real 1788204506] req@ffff8e82894e6d80 x1875062187965440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204506.629091:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187965440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204506.629099:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204506.629102:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204506.629105:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187965440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204507.332916:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204508.356524:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204509.380460:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204510.404504:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204511.428478:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204511.748409:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204511.748422:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204511.748426:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204511.748438:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5966 00000100:00080000:3.0:1788204511.748450:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204511.748457:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204511.748499:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187965568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204511.748523:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204511.748527:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204511.748528:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204511.748531:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5966 00000100:00080000:3.0:1788204511.748534:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204511.748537:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204511.748551:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204511.748568:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187965568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204511.748568:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187965696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204511.748597:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204511.748601:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204511.748603:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204511.748608:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5966 00000100:00080000:3.0:1788204511.748612:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204511.748616:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204511.748622:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204511.748635:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187965824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204511.748645:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204511.748649:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204511.748651:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788204511.748663:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788204511.748667:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5966 00000100:00080000:3.0:1788204511.748672:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204511.748678:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204511.748684:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204511.748699:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187965952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204511.748709:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204511.748713:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788204511.748716:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204511/real 1788204511] req@ffff8e828953a680 x1875062187965568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204511.748716:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204511.748725:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5966 00000100:00080000:3.0:1788204511.748731:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204511.748736:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204511.748738:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187965568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788204511.748743:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788204511.748749:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204511.748753:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204511.748759:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187965568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204511.748760:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187966080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204511.748783:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187965696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204511.748800:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204511.748808:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204511/real 1788204511] req@ffff8e8289539f80 x1875062187965696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204511.748816:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187965696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204511.748822:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204511.748824:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204511.748826:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187965696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204511.748830:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187965824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204511.748841:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204511.748846:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204511/real 1788204511] req@ffff8e8289538380 x1875062187965824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204511.748853:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187965824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204511.748857:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204511.748859:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204511.748861:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187965824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204511.748864:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187965952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204511.748874:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204511.748879:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204511/real 1788204511] req@ffff8e8289539500 x1875062187965952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204511.748886:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187965952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204511.748891:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204511.748892:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204511.748894:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187965952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204511.748897:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187966080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204511.748905:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204511.748911:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204511/real 1788204511] req@ffff8e8289539880 x1875062187966080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204511.748918:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187966080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204511.748922:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204511.748923:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204511.748925:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187966080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204512.452510:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204513.476580:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204514.500545:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204515.524415:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204516.548525:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204516.868447:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204516.868458:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204516.868463:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204516.868474:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5971 00000100:00080000:3.0:1788204516.868511:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204516.868521:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204516.868569:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187966208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204516.868595:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204516.868600:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204516.868602:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204516.868606:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5971 00000100:00080000:3.0:1788204516.868612:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204516.868617:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204516.868637:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204516.868647:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187966208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204516.868655:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187966336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204516.868664:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204516.868666:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204516.868667:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204516.868670:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5971 00000100:00080000:3.0:1788204516.868673:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204516.868676:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204516.868679:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204516.868687:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187966464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204516.868693:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204516.868695:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204516.868696:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204516.868698:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5971 00000100:00080000:3.0:1788204516.868724:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204516.868727:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204516.868730:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204516.868737:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187966592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204516.868750:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788204516.868752:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788204516.868753:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204516.868757:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204516.868760:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5971 00000100:00080000:3.0:1788204516.868762:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204516.868764:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204516.868767:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204516.868773:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187966720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204516.868819:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204516/real 1788204516] req@ffff8e828953b100 x1875062187966208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204516.868842:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187966208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204516.868854:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204516.868859:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204516.868863:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187966208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204516.868891:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187966336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204516.868917:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204516.868931:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204516/real 1788204516] req@ffff8e8289538a80 x1875062187966336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204516.868944:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187966336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204516.868959:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204516.868962:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204516.868966:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187966336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204516.868972:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187966464:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204516.868993:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204516.869005:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204516/real 1788204516] req@ffff8e828953bb80 x1875062187966464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204516.869018:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187966464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204516.869026:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204516.869029:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204516.869032:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187966464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204516.869038:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187966592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204516.869060:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204516.869072:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204516/real 1788204516] req@ffff8e8289539880 x1875062187966592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204516.869084:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187966592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204516.869092:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204516.869095:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204516.869099:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187966592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204516.869105:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187966720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204516.869125:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204516.869136:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204516/real 1788204516] req@ffff8e8289539500 x1875062187966720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204516.869147:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187966720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204516.869155:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204516.869158:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204516.869161:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187966720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204517.572553:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204518.596558:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204519.620558:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204520.644500:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204521.668565:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204521.989952:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204521.989963:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204521.989968:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204521.989979:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5977 00000100:00080000:0.0:1788204521.989990:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204521.989999:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204521.990048:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187966848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204521.990078:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204521.990086:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204521.990088:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204521.990093:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5977 00000100:00080000:0.0:1788204521.990098:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204521.990104:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204521.990116:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204521.990116:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187966848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204521.990144:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187966976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204521.990156:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204521.990160:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204521.990162:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204521.990166:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5977 00000100:00080000:0.0:1788204521.990170:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204521.990174:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204521.990179:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204521.990193:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187967104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204521.990203:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788204521.990204:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204521.990221:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204521.990223:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204521.990227:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5977 00000100:00080000:0.0:1788204521.990234:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204521.990239:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204521.990245:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204521.990258:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187967232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204521.990263:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204521/real 1788204521] req@ffff8e828793ca80 x1875062187966848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204521.990276:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204521.990283:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204521.990285:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204521.990293:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5977 00000100:00080000:0.0:1788204521.990299:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204521.990303:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204521.990333:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187966848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788204521.990339:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788204521.990350:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788204521.990356:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187967360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204521.990358:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204521.990367:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187966848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204521.990413:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187966976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204521.990438:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204521.990451:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204521/real 1788204521] req@ffff8e828793c000 x1875062187966976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204521.990463:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187966976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204521.990472:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204521.990475:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204521.990478:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187966976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204521.990484:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187967104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204521.990503:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204521.990513:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204521/real 1788204521] req@ffff8e828793dc00 x1875062187967104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204521.990525:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187967104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204521.990533:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204521.990536:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204521.990539:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187967104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204521.990544:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187967232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204521.990564:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204521.990575:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204521/real 1788204521] req@ffff8e828793f480 x1875062187967232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204521.990586:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187967232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204521.990594:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204521.990598:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204521.990601:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187967232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204521.990606:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187967360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204521.990625:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204521.990635:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204521/real 1788204521] req@ffff8e828793e300 x1875062187967360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204521.990646:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187967360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204521.990654:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204521.990657:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204521.990660:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187967360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204522.692454:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204523.716530:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204524.740480:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204525.764522:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204526.788510:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204527.108394:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204527.108402:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204527.108406:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204527.108414:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5982 00000100:00080000:0.0:1788204527.108424:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204527.108430:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204527.108467:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187967488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204527.108489:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204527.108491:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204527.108492:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204527.108495:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5982 00000100:00080000:0.0:1788204527.108497:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204527.108499:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204527.108507:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204527.108513:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187967616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204527.108519:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204527.108521:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204527.108523:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204527.108525:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5982 00000100:00080000:0.0:1788204527.108527:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204527.108529:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204527.108531:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204527.108536:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187967744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204527.108541:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204527.108543:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204527.108544:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204527.108547:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5982 00000100:00080000:0.0:1788204527.108551:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204527.108553:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204527.108556:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204527.108562:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187967872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204527.108562:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187967488:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204527.108575:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204527.108578:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204527.108579:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204527.108581:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5982 00000100:00080000:0.0:1788204527.108583:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204527.108585:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204527.108587:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204527.108595:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187968000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204527.108635:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204527.108677:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204527/real 1788204527] req@ffff8e828793d880 x1875062187967488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204527.108695:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187967488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204527.108704:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204527.108708:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204527.108712:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187967488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204527.108719:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187967616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204527.108741:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204527.108751:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204527/real 1788204527] req@ffff8e828793fb80 x1875062187967616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204527.108759:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187967616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204527.108764:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204527.108766:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204527.108768:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187967616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204527.108785:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187967744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204527.108796:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204527.108802:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204527/real 1788204527] req@ffff8e828793d500 x1875062187967744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204527.108808:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187967744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204527.108813:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204527.108814:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204527.108816:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187967744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204527.108818:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187967872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204527.108827:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204527.108832:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204527/real 1788204527] req@ffff8e828793c380 x1875062187967872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204527.108839:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187967872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204527.108843:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204527.108845:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204527.108846:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187967872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204527.108849:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187968000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204527.108866:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204527.108873:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204527/real 1788204527] req@ffff8e828793e300 x1875062187968000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204527.108879:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187968000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204527.108885:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204527.108888:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204527.108891:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187968000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204527.812484:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204528.836540:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204529.860534:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204530.884527:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204531.908502:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204532.228475:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204532.228489:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204532.228495:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204532.228508:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5987 00000100:00080000:3.0:1788204532.228521:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204532.228544:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204532.228592:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187968128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204532.228623:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204532.228628:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204532.228630:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204532.228634:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5987 00000100:00080000:3.0:1788204532.228638:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204532.228642:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204532.228654:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204532.228671:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187968256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204532.228682:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204532.228686:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788204532.228690:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187968128:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204532.228690:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204532.228696:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5987 00000100:00080000:3.0:1788204532.228701:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204532.228705:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204532.228711:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204532.228723:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187968384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204532.228732:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204532.228736:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204532.228737:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204532.228740:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5987 00000100:00080000:3.0:1788204532.228755:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204532.228760:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204532.228765:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788204532.228773:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204532.228778:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187968512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204532.228787:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204532.228791:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204532.228794:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204532.228798:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5987 00000100:00080000:3.0:1788204532.228802:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204532.228806:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204532.228811:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204532.228823:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204532/real 1788204532] req@ffff8e8289538380 x1875062187968128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204532.228825:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187968640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204532.228866:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187968128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204532.228881:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204532.228885:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204532.228889:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187968128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204532.228916:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187968256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204532.228939:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204532.228952:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204532/real 1788204532] req@ffff8e8289539f80 x1875062187968256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204532.228965:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187968256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204532.228973:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204532.228976:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204532.228978:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187968256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204532.228982:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187968384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204532.229000:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204532.229010:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204532/real 1788204532] req@ffff8e828953a680 x1875062187968384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204532.229021:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187968384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204532.229030:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204532.229033:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204532.229036:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187968384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204532.229041:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187968512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204532.229057:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204532.229063:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204532/real 1788204532] req@ffff8e8289539500 x1875062187968512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204532.229074:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187968512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204532.229082:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204532.229085:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204532.229089:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187968512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204532.229094:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187968640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204532.229113:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204532.229123:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204532/real 1788204532] req@ffff8e8289539880 x1875062187968640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204532.229131:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187968640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204532.229138:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204532.229141:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204532.229144:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187968640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204532.932511:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204533.956506:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204534.980522:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204536.004510:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204537.028538:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204537.348416:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204537.348429:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204537.348433:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204537.348445:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5992 00000100:00080000:0.0:1788204537.348458:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204537.348467:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204537.348512:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187968768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204537.348539:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204537.348546:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204537.348548:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204537.348557:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5992 00000100:00080000:0.0:1788204537.348562:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204537.348567:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204537.348584:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204537.348589:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187968768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204537.348602:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187968896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204537.348620:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204537.348623:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204537.348625:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204537.348629:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5992 00000100:00080000:0.0:1788204537.348634:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204537.348638:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204537.348644:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204537.348656:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187969024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204537.348665:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204537.348669:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204537.348671:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788204537.348676:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204537.348677:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5992 00000100:00080000:0.0:1788204537.348683:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204537.348687:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204537.348693:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204537.348704:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187969152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204537.348713:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204537.348716:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204537.348718:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204537.348722:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5992 00000100:00080000:0.0:1788204537.348726:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788204537.348732:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204537/real 1788204537] req@ffff8e828793f480 x1875062187968768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204537.348733:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204537.348741:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204537.348754:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187968768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204537.348758:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187969280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204537.348778:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204537.348785:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204537.348790:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187968768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204537.348815:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187968896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204537.348833:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204537.348841:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204537/real 1788204537] req@ffff8e828793dc00 x1875062187968896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204537.348849:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187968896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204537.348855:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204537.348857:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204537.348859:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187968896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204537.348862:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187969024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204537.348871:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204537.348877:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204537/real 1788204537] req@ffff8e828793c000 x1875062187969024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204537.348883:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187969024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204537.348892:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204537.348894:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204537.348896:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187969024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204537.348899:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187969152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204537.348915:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204537.348925:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204537/real 1788204537] req@ffff8e828793ca80 x1875062187969152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204537.348936:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187969152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204537.348944:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204537.348947:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204537.348950:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187969152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204537.348954:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187969280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204537.348969:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204537.348975:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204537/real 1788204537] req@ffff8e828793f800 x1875062187969280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204537.348982:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187969280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204537.348987:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204537.348989:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204537.348992:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187969280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204538.052526:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204539.076492:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204540.100508:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204541.124556:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204542.148457:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204542.468420:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204542.468434:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204542.468439:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204542.468452:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 5997 00000100:00080000:3.0:1788204542.468467:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204542.468476:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204542.468525:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187969408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204542.468556:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204542.468559:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204542.468561:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204542.468564:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5997 00000100:00080000:3.0:1788204542.468569:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204542.468573:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204542.468584:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204542.468602:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187969408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204542.468603:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187969536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204542.468624:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204542.468636:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204542.468638:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204542.468642:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5997 00000100:00080000:3.0:1788204542.468646:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204542.468650:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204542.468655:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204542.468666:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187969664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204542.468675:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204542.468677:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204542.468678:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204542.468680:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5997 00000100:00080000:3.0:1788204542.468683:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204542.468686:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204542.468690:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788204542.468691:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204542.468711:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187969792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204542.468722:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204542.468725:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204542.468726:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204542.468730:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 5997 00000100:00080000:3.0:1788204542.468735:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204542.468740:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204542.468745:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204542.468753:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204542/real 1788204542] req@ffff8e828953bb80 x1875062187969408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204542.468760:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187969920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204542.468782:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187969408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204542.468795:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204542.468799:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204542.468804:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187969408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204542.468833:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187969536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204542.468860:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204542.468877:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204542/real 1788204542] req@ffff8e8289538a80 x1875062187969536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204542.468898:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187969536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204542.468907:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204542.468911:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204542.468914:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187969536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204542.468920:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187969664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204542.468939:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204542.468949:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204542/real 1788204542] req@ffff8e828953b100 x1875062187969664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204542.468957:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187969664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204542.468962:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204542.468965:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204542.468968:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187969664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204542.468974:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187969792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204542.468996:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204542.469018:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204542/real 1788204542] req@ffff8e8289539880 x1875062187969792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204542.469032:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187969792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204542.469051:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204542.469055:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204542.469059:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187969792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204542.469066:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187969920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204542.469083:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204542.469091:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204542/real 1788204542] req@ffff8e8289539500 x1875062187969920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204542.469099:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187969920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204542.469105:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204542.469107:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204542.469108:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187969920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204543.172539:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204544.196647:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204545.220510:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204546.244550:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204547.268496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204547.588464:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204547.588476:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204547.588481:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204547.588493:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6002 00000100:00080000:3.0:1788204547.588505:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204547.588512:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204547.588561:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187970048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204547.588595:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204547.588599:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204547.588601:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204547.588606:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6002 00000100:00080000:3.0:1788204547.588610:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204547.588615:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204547.588627:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204547.588643:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187970176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204547.588651:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204547.588656:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788204547.588657:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187970048:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204547.588658:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204547.588665:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6002 00000100:00080000:3.0:1788204547.588668:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204547.588671:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204547.588674:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204547.588681:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187970304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204547.588687:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204547.588689:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204547.588690:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204547.588692:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6002 00000100:00080000:3.0:1788204547.588694:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204547.588696:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204547.588700:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204547.588706:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187970432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204547.588711:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204547.588714:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204547.588715:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204547.588717:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6002 00000100:00080000:3.0:1788204547.588724:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204547.588726:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204547.588729:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204547.588736:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187970560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204547.588742:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204547.588800:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204547/real 1788204547] req@ffff8e828953a680 x1875062187970048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204547.588815:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187970048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204547.588824:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204547.588828:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204547.588831:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187970048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204547.588853:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187970176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204547.588873:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204547.588885:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204547/real 1788204547] req@ffff8e8289539f80 x1875062187970176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204547.588897:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187970176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204547.588905:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204547.588908:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204547.588912:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187970176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204547.588917:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187970304:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204547.588932:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204547.588940:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204547/real 1788204547] req@ffff8e8289538380 x1875062187970304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204547.588947:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187970304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204547.588973:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204547.588975:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204547.588978:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187970304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204547.588982:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187970432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204547.588996:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204547.589003:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204547/real 1788204547] req@ffff8e8289539500 x1875062187970432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204547.589010:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187970432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204547.589015:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204547.589017:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204547.589019:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187970432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204547.589021:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187970560:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204547.589031:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204547.589036:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204547/real 1788204547] req@ffff8e8289539880 x1875062187970560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204547.589045:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187970560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204547.589053:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204547.589056:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204547.589059:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187970560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204548.292508:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204549.316513:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204550.340504:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204551.364471:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204552.388503:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204552.708386:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204552.708395:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204552.708399:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204552.708409:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6007 00000100:00080000:3.0:1788204552.708421:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204552.708428:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204552.708463:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062187970688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204552.708489:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204552.708493:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204552.708494:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204552.708498:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6007 00000100:00080000:3.0:1788204552.708502:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204552.708507:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204552.708517:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204552.708532:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062187970816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204552.708536:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187970688:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788204552.708550:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204552.708553:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204552.708556:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204552.708560:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6007 00000100:00080000:3.0:1788204552.708565:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204552.708569:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204552.708574:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204552.708583:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062187970944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204552.708590:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204552.708593:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1788204552.708594:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788204552.708595:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204552.708601:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6007 00000100:00080000:3.0:1788204552.708604:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204552.708607:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204552.708611:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204552.708619:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187971072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204552.708626:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204552.708630:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204552.708631:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204552.708635:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6007 00000100:00080000:3.0:1788204552.708638:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204552.708642:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204552.708647:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204552.708650:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204552/real 1788204552] req@ffff8e828953b100 x1875062187970688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204552.708660:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187971200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204552.708672:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062187970688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204552.708687:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204552.708692:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204552.708696:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187970688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204552.708719:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187970816:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204552.708741:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204552.708754:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204552/real 1788204552] req@ffff8e8289538a80 x1875062187970816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204552.708766:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062187970816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204552.708780:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204552.708783:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204552.708786:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187970816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204552.708792:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187970944:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204552.708812:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204552.708823:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204552/real 1788204552] req@ffff8e828953bb80 x1875062187970944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204552.708834:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062187970944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204552.708849:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204552.708853:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204552.708856:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187970944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204552.708872:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187971072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204552.708893:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204552.708904:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204552/real 1788204552] req@ffff8e8289539880 x1875062187971072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204552.708916:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187971072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204552.708923:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204552.708926:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204552.708929:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187971072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204552.708935:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187971200:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204552.708954:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204552.708964:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204552/real 1788204552] req@ffff8e8289539500 x1875062187971200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204552.708975:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187971200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204552.708982:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204552.708985:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204552.708988:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187971200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204553.412535:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204554.436679:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204555.460526:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204556.484500:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204557.508514:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204557.828485:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204557.828496:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204557.828500:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204557.828511:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6012 00000100:00080000:2.0:1788204557.828524:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204557.828531:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204557.828572:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187971328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204557.828596:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204557.828599:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204557.828600:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204557.828603:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6012 00000100:00080000:2.0:1788204557.828605:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204557.828609:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204557.828618:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204557.828627:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187971456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204557.828634:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204557.828636:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204557.828639:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204557.828642:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6012 00000100:00100000:3.0:1788204557.828642:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187971328:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204557.828644:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204557.828649:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204557.828654:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204557.828660:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187971584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204557.828665:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204557.828667:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204557.828669:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204557.828671:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6012 00000100:00080000:2.0:1788204557.828673:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204557.828675:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204557.828678:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204557.828685:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187971712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204557.828690:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204557.828693:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204557.828694:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204557.828696:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6012 00000100:00080000:2.0:1788204557.828710:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204557.828712:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204557.828715:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204557.828722:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187971840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204557.828730:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204557.828792:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204557/real 1788204557] req@ffff8e82894e4e00 x1875062187971328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204557.828807:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187971328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204557.828815:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204557.828818:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204557.828821:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187971328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204557.828842:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187971456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204557.828858:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204557.828866:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204557/real 1788204557] req@ffff8e82894e4a80 x1875062187971456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204557.828873:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187971456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204557.828877:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204557.828879:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204557.828881:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187971456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204557.828884:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187971584:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204557.828896:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204557.828902:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204557/real 1788204557] req@ffff8e82894e6300 x1875062187971584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204557.828908:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187971584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204557.828913:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204557.828914:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204557.828916:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187971584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204557.828919:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187971712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204557.828932:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204557.828942:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204557/real 1788204557] req@ffff8e82894e7800 x1875062187971712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204557.828952:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187971712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204557.828959:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204557.828963:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204557.828966:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187971712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204557.828971:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187971840:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204557.828987:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204557.829001:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204557/real 1788204557] req@ffff8e82894e4000 x1875062187971840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204557.829008:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187971840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204557.829013:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204557.829015:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204557.829017:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187971840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204558.532519:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204559.556466:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204560.580496:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204561.604504:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204562.628486:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204562.948455:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204562.948465:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204562.948469:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204562.948479:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6018 00000100:00080000:1.0:1788204562.948489:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204562.948496:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204562.948531:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187971968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204562.948551:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204562.948554:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204562.948555:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204562.948558:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6018 00000100:00080000:1.0:1788204562.948560:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204562.948562:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204562.948569:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204562.948578:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187972096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204562.948584:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204562.948587:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204562.948588:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204562.948590:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6018 00000100:00080000:1.0:1788204562.948592:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204562.948594:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204562.948596:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204562.948602:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187972224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204562.948613:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187971968:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204562.948616:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204562.948623:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204562.948627:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204562.948639:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6018 00000100:00080000:1.0:1788204562.948641:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204562.948644:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204562.948647:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204562.948659:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187972352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204562.948665:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204562.948667:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204562.948668:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204562.948670:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6018 00000100:00080000:1.0:1788204562.948672:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204562.948674:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204562.948677:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204562.948695:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187972480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204562.948738:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204562.948795:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204562/real 1788204562] req@ffff8e828d720a80 x1875062187971968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204562.948815:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187971968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204562.948826:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204562.948830:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204562.948835:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187971968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204562.948843:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187972096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204562.948864:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204562.948875:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204562/real 1788204562] req@ffff8e828d721180 x1875062187972096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204562.948885:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187972096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204562.948893:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204562.948896:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204562.948900:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187972096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204562.948930:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187972224:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204562.948947:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204562.948956:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204562/real 1788204562] req@ffff8e828d722d80 x1875062187972224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204562.948967:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187972224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204562.948975:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204562.948978:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204562.948981:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187972224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204562.948986:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187972352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204562.949012:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204562.949021:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204562/real 1788204562] req@ffff8e828d720380 x1875062187972352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204562.949031:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187972352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204562.949039:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204562.949042:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204562.949045:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187972352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204562.949050:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187972480:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204562.949067:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204562.949075:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204562/real 1788204562] req@ffff8e828d722300 x1875062187972480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204562.949085:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187972480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204562.949092:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204562.949095:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204562.949098:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187972480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204563.652501:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204564.676477:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204565.700511:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204566.724464:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204567.748470:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204568.068400:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204568.068410:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204568.068414:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204568.068423:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6023 00000100:00080000:2.0:1788204568.068433:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204568.068439:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204568.068474:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187972608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204568.068501:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204568.068505:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204568.068506:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204568.068509:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6023 00000100:00080000:2.0:1788204568.068512:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204568.068516:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204568.068523:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204568.068535:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187972736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204568.068545:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204568.068550:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788204568.068551:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187972608:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204568.068552:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204568.068560:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6023 00000100:00080000:2.0:1788204568.068563:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204568.068566:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204568.068570:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204568.068582:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187972864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204568.068590:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204568.068618:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204568.068620:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204568.068623:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6023 00000100:00080000:2.0:1788204568.068627:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204568.068631:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204568.068635:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204568.068646:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187972992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204568.068655:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204568.068659:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204568.068661:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204568.068665:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6023 00000100:00080000:2.0:1788204568.068668:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204568.068669:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187972736:192.168.201.121@tcp:38:kworker.0 00000100:00080000:2.0:1788204568.068677:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204568.068682:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204568.068693:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187973120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204568.068705:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187972864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204568.068723:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187972992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204568.068736:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187973120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204568.772448:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204569.796472:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204570.820429:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788204571.140643:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:0.0:1788204571.140722:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788204571.140748:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:0.0:1788204571.140797:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:0.0:1788204571.140809:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:0.0:1788204571.140819:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:0.0:1788204571.140826:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:3.0:1788204571.140906:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204568/real 1788204571] req@ffff8e82894e4000 x1875062187973120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204571.140941:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187973120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204571.140953:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204571.140958:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204571.140964:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187973120:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204571.140971:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204568/real 1788204571] req@ffff8e82894e7480 x1875062187972736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204571.140989:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187972736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204571.140997:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204571.141000:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204571.141003:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187972736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204571.141010:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204568/real 1788204571] req@ffff8e82894e6a00 x1875062187972864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204571.141021:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187972864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204571.141028:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204571.141032:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204571.141035:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187972864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204571.141049:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204568/real 1788204571] req@ffff8e82894e5880 x1875062187972992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204571.141060:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187972992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204571.141067:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204571.141069:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204571.141072:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187972992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204571.141089:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204568/real 1788204571] req@ffff8e82894e6680 x1875062187972608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204571.141101:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187972608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204571.141107:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204571.141110:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204571.141112:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187972608:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788204571.844497:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204572.868498:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204573.188421:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204573.188435:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204573.188441:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204573.188453:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6028 00000100:00080000:2.0:1788204573.188466:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204573.188474:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204573.188550:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187973248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204573.188586:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204573.188591:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204573.188593:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204573.188597:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6028 00000100:00080000:2.0:1788204573.188601:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204573.188605:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204573.188617:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204573.188630:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187973376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204573.188640:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204573.188646:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204573.188647:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204573.188651:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6028 00000100:00080000:2.0:1788204573.188654:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204573.188658:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204573.188663:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204573.188672:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187973504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204573.188682:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204573.188685:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204573.188686:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204573.188690:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6028 00000100:00080000:2.0:1788204573.188693:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204573.188697:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204573.188702:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204573.188711:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187973632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204573.188719:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204573.188723:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204573.188725:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204573.188729:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6028 00000100:00080000:2.0:1788204573.188732:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204573.188736:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204573.188741:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204573.188751:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187973760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204573.188854:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187973248:192.168.201.121@tcp:250:kworker.0 00000800:00000100:3.0:1788204573.188969:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204573.189043:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204573/real 1788204573] req@ffff8e82894e7800 x1875062187973248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204573.189062:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187973248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204573.189073:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204573.189078:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204573.189082:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187973248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204573.189089:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187973376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204573.189187:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204573.189201:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204573/real 1788204573] req@ffff8e82894e6300 x1875062187973376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204573.189213:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187973376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204573.189221:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204573.189225:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204573.189228:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187973376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204573.189234:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187973504:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204573.189253:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204573.189264:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204573/real 1788204573] req@ffff8e82894e4a80 x1875062187973504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204573.189275:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187973504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204573.189282:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204573.189286:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204573.189289:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187973504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204573.189294:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187973632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204573.189346:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204573.189356:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204573/real 1788204573] req@ffff8e82894e4e00 x1875062187973632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204573.189368:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187973632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204573.189375:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204573.189378:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204573.189381:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187973632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204573.189386:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187973760:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204573.189405:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204573.189416:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204573/real 1788204573] req@ffff8e82894e6d80 x1875062187973760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204573.189429:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187973760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204573.189437:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204573.189441:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204573.189446:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187973760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204573.892483:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204574.916449:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204575.940469:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204576.964481:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204577.988450:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204578.308472:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204578.308487:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204578.308493:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204578.308512:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6033 00000100:00080000:2.0:1788204578.308542:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204578.308552:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204578.308596:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187973888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204578.308623:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204578.308627:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204578.308629:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204578.308634:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6033 00000100:00080000:2.0:1788204578.308637:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204578.308645:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204578.308645:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187973888:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788204578.308660:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204578.308677:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187974016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204578.308688:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204578.308692:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204578.308693:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204578.308696:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6033 00000100:00080000:2.0:1788204578.308701:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204578.308705:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204578.308710:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788204578.308717:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204578.308723:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187974144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204578.308740:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204578.308743:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204578.308745:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204578.308748:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6033 00000100:00080000:2.0:1788204578.308751:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204578.308755:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204578.308759:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204578.308771:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187974272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204578.308779:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204578.308783:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204578.308785:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204578.308788:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6033 00000100:00080000:2.0:1788204578.308791:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204578.308796:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204578.308801:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204578.308812:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187974400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204578.308826:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204578/real 1788204578] req@ffff8e82894e5f80 x1875062187973888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204578.308937:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187973888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204578.308946:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204578.308950:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204578.308987:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187973888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204578.309012:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187974016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204578.309030:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204578.309039:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204578/real 1788204578] req@ffff8e82894e4380 x1875062187974016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204578.309051:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187974016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204578.309057:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204578.309059:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204578.309061:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187974016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204578.309064:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187974144:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204578.309075:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204578.309083:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204578/real 1788204578] req@ffff8e82894e5c00 x1875062187974144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204578.309091:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187974144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204578.309096:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204578.309098:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204578.309100:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187974144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204578.309103:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187974272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204578.309115:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204578.309120:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204578/real 1788204578] req@ffff8e82894e7100 x1875062187974272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204578.309126:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187974272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204578.309135:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204578.309137:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204578.309139:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187974272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204578.309142:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187974400:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204578.309156:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204578.309162:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204578/real 1788204578] req@ffff8e82894e6d80 x1875062187974400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204578.309168:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187974400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204578.309172:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204578.309174:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204578.309175:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187974400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204579.012446:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204580.036508:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204581.060471:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204582.084492:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204583.108492:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204583.428433:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204583.428443:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204583.428447:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204583.428457:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6038 00000100:00080000:1.0:1788204583.428467:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204583.428473:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204583.428511:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187974528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204583.428537:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204583.428540:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204583.428542:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204583.428547:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6038 00000100:00080000:1.0:1788204583.428551:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204583.428556:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204583.428566:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204583.428572:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187974528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204583.428590:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187974656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204583.428602:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204583.428607:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204583.428609:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204583.428613:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6038 00000100:00080000:1.0:1788204583.428618:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788204583.428623:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788204583.428624:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204583.428631:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204583.428644:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187974784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204583.428661:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204583.428665:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204583.428668:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788204583.428669:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204583/real 1788204583] req@ffff8e828d723800 x1875062187974528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204583.428674:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6038 00000100:00080000:1.0:1788204583.428681:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204583.428689:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204583.428694:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204583.428696:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187974528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204583.428707:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1788204583.428709:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187974912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204583.428712:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204583.428719:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187974528:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204583.428730:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204583.428734:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204583.428736:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204583.428740:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6038 00000100:00100000:3.0:1788204583.428744:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187974656:192.168.201.121@tcp:38:kworker.0 00000100:00080000:1.0:1788204583.428745:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204583.428752:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204583.428759:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788204583.428764:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204583.428773:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187975040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204583.428776:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204583/real 1788204583] req@ffff8e828d721500 x1875062187974656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204583.428801:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187974656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204583.428807:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204583.428809:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204583.428812:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187974656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204583.428817:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187974784:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204583.428827:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204583.428833:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204583/real 1788204583] req@ffff8e828d720000 x1875062187974784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204583.428839:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187974784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204583.428844:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204583.428845:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204583.428847:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187974784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204583.428850:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187974912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204583.428860:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204583.428865:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204583/real 1788204583] req@ffff8e828d722680 x1875062187974912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204583.428872:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187974912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204583.428876:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204583.428878:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204583.428879:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187974912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204583.428889:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187975040:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204583.428898:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204583.428903:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204583/real 1788204583] req@ffff8e828d720e00 x1875062187975040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204583.428909:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187975040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204583.428913:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204583.428915:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204583.428916:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187975040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204584.132505:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204585.156546:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204586.180447:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204587.204527:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204588.228461:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204588.548403:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204588.548411:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204588.548414:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204588.548422:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6043 00000100:00080000:0.0:1788204588.548430:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204588.548435:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204588.548462:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187975168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204588.548482:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204588.548484:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204588.548485:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204588.548488:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6043 00000100:00080000:0.0:1788204588.548490:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204588.548492:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204588.548498:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204588.548505:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187975296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204588.548511:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204588.548513:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204588.548515:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204588.548517:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6043 00000100:00080000:0.0:1788204588.548519:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204588.548521:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204588.548524:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204588.548529:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187975168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204588.548531:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187975424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204588.548540:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204588.548542:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204588.548543:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204588.548546:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6043 00000100:00080000:0.0:1788204588.548548:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204588.548550:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204588.548552:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204588.548560:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187975552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204588.548565:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204588.548567:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204588.548568:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204588.548571:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6043 00000100:00080000:0.0:1788204588.548577:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204588.548580:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:3.0:1788204588.548585:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788204588.548586:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204588.548596:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187975680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204588.548618:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204588/real 1788204588] req@ffff8e828793e680 x1875062187975168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204588.548630:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187975168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204588.548641:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204588.548643:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204588.548645:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187975168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204588.548650:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187975296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204588.548662:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204588.548668:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204588/real 1788204588] req@ffff8e828793c700 x1875062187975296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204588.548675:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187975296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204588.548679:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204588.548681:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204588.548682:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187975296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204588.548697:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187975424:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204588.548707:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204588.548712:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204588/real 1788204588] req@ffff8e828793f100 x1875062187975424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204588.548718:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187975424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204588.548722:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204588.548724:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204588.548725:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187975424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204588.548729:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187975552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204588.548738:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204588.548743:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204588/real 1788204588] req@ffff8e828793ce00 x1875062187975552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204588.548749:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187975552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204588.548753:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204588.548755:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204588.548756:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187975552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204588.548759:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187975680:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204588.548767:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204588.548771:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204588/real 1788204588] req@ffff8e828793f800 x1875062187975680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204588.548777:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187975680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204588.548781:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204588.548783:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204588.548784:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187975680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204589.252551:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204590.276481:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204591.300516:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204592.324476:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204593.348505:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204593.668472:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204593.668481:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204593.668486:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204593.668496:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6048 00000100:00080000:0.0:1788204593.668507:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204593.668515:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204593.668563:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187975808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204593.668592:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204593.668595:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204593.668597:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204593.668600:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6048 00000100:00080000:0.0:1788204593.668603:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204593.668606:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204593.668615:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204593.668625:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187975936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204593.668631:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204593.668634:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204593.668635:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204593.668637:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6048 00000100:00080000:0.0:1788204593.668640:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204593.668645:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204593.668649:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204593.668660:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187976064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204593.668669:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204593.668672:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204593.668674:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204593.668678:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6048 00000100:00080000:0.0:1788204593.668682:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204593.668689:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204593.668689:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187975808:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788204593.668693:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204593.668703:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187976192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204593.668710:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204593.668712:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204593.668714:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204593.668716:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6048 00000100:00080000:0.0:1788204593.668718:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204593.668721:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204593.668723:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204593.668731:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187976320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204593.668773:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204593.668827:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204593/real 1788204593] req@ffff8e828793ca80 x1875062187975808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204593.668846:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187975808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204593.668856:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204593.668860:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204593.668864:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187975808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204593.668872:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187975936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204593.668891:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204593.668902:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204593/real 1788204593] req@ffff8e828793c000 x1875062187975936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204593.668913:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187975936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204593.668921:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204593.668923:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204593.668927:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187975936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204593.668931:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187976064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204593.668946:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204593.668955:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204593/real 1788204593] req@ffff8e828793dc00 x1875062187976064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204593.668966:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187976064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204593.668979:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204593.668982:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204593.668985:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187976064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204593.669011:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187976192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204593.669029:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204593.669038:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204593/real 1788204593] req@ffff8e828793f480 x1875062187976192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204593.669048:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187976192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204593.669056:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204593.669059:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204593.669062:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187976192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204593.669067:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187976320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204593.669082:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204593.669091:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204593/real 1788204593] req@ffff8e828793e300 x1875062187976320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204593.669102:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187976320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204593.669109:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204593.669111:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204593.669115:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187976320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204594.372504:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204595.396508:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204596.420451:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204597.446154:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204598.468451:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204598.788441:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204598.788451:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204598.788456:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204598.788467:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6053 00000100:00080000:0.0:1788204598.788479:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204598.788485:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204598.788526:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187976448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204598.788551:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204598.788555:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204598.788556:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204598.788559:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6053 00000100:00080000:0.0:1788204598.788561:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204598.788563:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204598.788573:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204598.788585:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187976576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204598.788596:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187976448:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204598.788597:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204598.788606:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204598.788608:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204598.788612:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6053 00000100:00080000:0.0:1788204598.788616:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204598.788621:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204598.788626:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204598.788639:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187976704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204598.788647:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204598.788649:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204598.788651:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204598.788654:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6053 00000100:00080000:0.0:1788204598.788658:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204598.788661:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204598.788666:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788204598.788669:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204598.788678:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187976832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204598.788686:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204598.788688:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204598.788690:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204598.788691:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6053 00000100:00080000:0.0:1788204598.788700:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204598.788702:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204598.788705:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204598.788713:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187976960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204598.788726:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204598/real 1788204598] req@ffff8e828793c380 x1875062187976448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204598.788747:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187976448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204598.788759:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204598.788764:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204598.788769:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187976448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204598.788796:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187976576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204598.788819:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204598.788833:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204598/real 1788204598] req@ffff8e828793d500 x1875062187976576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204598.788852:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187976576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204598.788858:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204598.788863:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204598.788866:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187976576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204598.788870:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187976704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204598.788884:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204598.788891:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204598/real 1788204598] req@ffff8e828793fb80 x1875062187976704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204598.788899:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187976704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204598.788904:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204598.788905:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204598.788907:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187976704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204598.788910:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187976832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204598.788920:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204598.788925:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204598/real 1788204598] req@ffff8e828793d880 x1875062187976832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204598.788932:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187976832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204598.788936:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204598.788938:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204598.788939:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187976832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204598.788943:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187976960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204598.788959:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204598.788968:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204598/real 1788204598] req@ffff8e828793e300 x1875062187976960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204598.788979:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187976960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204598.788988:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204598.788991:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204598.788994:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187976960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204599.492546:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204600.516433:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204601.540503:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204602.564477:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204603.588477:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204603.908433:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204603.908447:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204603.908452:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204603.908465:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6058 00000100:00080000:0.0:1788204603.908477:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204603.908484:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204603.908523:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187977088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204603.908562:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204603.908566:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204603.908568:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204603.908572:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6058 00000100:00080000:0.0:1788204603.908575:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204603.908578:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204603.908588:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204603.908602:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187977216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204603.908612:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204603.908616:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204603.908620:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788204603.908621:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187977088:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204603.908626:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6058 00000100:00080000:0.0:1788204603.908633:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204603.908637:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204603.908642:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204603.908656:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187977344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204603.908667:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204603.908670:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204603.908672:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204603.908675:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6058 00000100:00080000:0.0:1788204603.908678:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204603.908682:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204603.908686:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204603.908694:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187977472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204603.908705:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204603.908708:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204603.908712:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788204603.908713:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204603.908719:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6058 00000100:00080000:0.0:1788204603.908723:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204603.908727:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204603.908732:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204603.908746:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187977600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204603.908771:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204603/real 1788204603] req@ffff8e828793f480 x1875062187977088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204603.908790:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187977088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204603.908801:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204603.908805:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204603.908809:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187977088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204603.908834:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187977216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204603.908855:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204603.908867:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204603/real 1788204603] req@ffff8e828793dc00 x1875062187977216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204603.908877:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187977216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204603.908885:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204603.908888:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204603.908892:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187977216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204603.908898:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187977344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204603.908917:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204603.908927:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204603/real 1788204603] req@ffff8e828793c000 x1875062187977344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204603.908939:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187977344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204603.908946:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204603.908949:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204603.908952:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187977344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204603.908958:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187977472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204603.909003:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204603.909014:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204603/real 1788204603] req@ffff8e828793ca80 x1875062187977472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204603.909025:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187977472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204603.909033:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204603.909036:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204603.909039:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187977472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204603.909044:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187977600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204603.909062:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204603.909072:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204603/real 1788204603] req@ffff8e828793f800 x1875062187977600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204603.909083:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187977600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204603.909090:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204603.909093:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204603.909097:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187977600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204604.612507:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204605.636743:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204606.660510:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204607.684446:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204608.708488:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204609.028430:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204609.028446:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204609.028450:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204609.028460:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6064 00000100:00080000:0.0:1788204609.028471:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204609.028481:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204609.028520:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187977728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204609.028544:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204609.028547:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204609.028548:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204609.028552:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6064 00000100:00080000:0.0:1788204609.028554:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204609.028556:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204609.028565:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204609.028573:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187977856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204609.028581:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204609.028583:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204609.028584:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204609.028587:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6064 00000100:00080000:0.0:1788204609.028588:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204609.028591:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204609.028593:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204609.028599:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187977984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204609.028605:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187977728:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204609.028607:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204609.028614:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204609.028620:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204609.028624:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6064 00000100:00080000:0.0:1788204609.028627:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204609.028632:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204609.028637:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204609.028651:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187978112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204609.028661:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204609.028664:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204609.028666:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204609.028668:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6064 00000100:00080000:0.0:1788204609.028678:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204609.028682:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204609.028686:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788204609.028691:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204609.028700:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187978240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204609.028747:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204609/real 1788204609] req@ffff8e828793ce00 x1875062187977728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204609.028767:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187977728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204609.028779:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204609.028783:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204609.028787:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187977728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204609.028795:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187977856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204609.028816:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204609.028828:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204609/real 1788204609] req@ffff8e828793f100 x1875062187977856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204609.028840:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187977856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204609.028855:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204609.028864:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204609.028868:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187977856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204609.028909:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187977984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204609.028928:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204609.028939:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204609/real 1788204609] req@ffff8e828793c700 x1875062187977984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204609.028951:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187977984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204609.028959:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204609.028962:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204609.028971:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187977984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204609.028978:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187978112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204609.028996:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204609.029006:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204609/real 1788204609] req@ffff8e828793e680 x1875062187978112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204609.029017:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187978112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204609.029024:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204609.029027:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204609.029031:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187978112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204609.029037:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187978240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204609.029056:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204609.029067:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204609/real 1788204609] req@ffff8e828793f800 x1875062187978240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204609.029078:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187978240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204609.029086:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204609.029089:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204609.029092:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187978240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204609.732478:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204610.756473:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204611.780495:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204612.804520:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204613.828491:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204614.148445:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204614.148460:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204614.148466:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204614.148478:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6069 00000100:00080000:0.0:1788204614.148492:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204614.148502:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204614.148547:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187978368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204614.148573:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204614.148577:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204614.148579:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204614.148583:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6069 00000100:00080000:0.0:1788204614.148587:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204614.148590:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204614.148599:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187978368:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788204614.148601:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204614.148615:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187978496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204614.148626:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204614.148628:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204614.148629:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204614.148632:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6069 00000100:00080000:0.0:1788204614.148635:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204614.148637:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204614.148640:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204614.148647:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187978624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204614.148653:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204614.148656:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204614.148657:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204614.148660:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6069 00000100:00080000:0.0:1788204614.148663:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204614.148665:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204614.148668:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204614.148676:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187978752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204614.148683:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204614.148687:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204614.148690:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204614.148691:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204614.148694:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6069 00000100:00080000:0.0:1788204614.148697:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204614.148700:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204614.148703:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204614.148709:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187978880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204614.148727:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204614/real 1788204614] req@ffff8e828793ca80 x1875062187978368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204614.148808:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187978368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204614.148818:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204614.148821:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204614.148824:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187978368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204614.148845:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187978496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204614.148869:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204614.148879:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204614/real 1788204614] req@ffff8e828793c000 x1875062187978496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204614.148886:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187978496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204614.148895:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204614.148897:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204614.148899:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187978496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204614.148905:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187978624:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204614.148923:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204614.148934:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204614/real 1788204614] req@ffff8e828793dc00 x1875062187978624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204614.149057:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187978624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204614.149063:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204614.149066:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204614.149069:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187978624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204614.149074:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187978752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204614.149139:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204614.149151:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204614/real 1788204614] req@ffff8e828793f480 x1875062187978752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204614.149159:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187978752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204614.149165:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204614.149167:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204614.149169:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187978752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204614.149172:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187978880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204614.149184:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204614.149190:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204614/real 1788204614] req@ffff8e828793e300 x1875062187978880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204614.149196:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187978880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204614.149200:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204614.149202:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204614.149203:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187978880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204614.852511:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204615.876484:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204616.900514:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204617.924484:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204618.948553:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204619.268442:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204619.268454:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204619.268459:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204619.268469:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6074 00000100:00080000:0.0:1788204619.268480:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204619.268488:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204619.268529:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187979008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204619.268553:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204619.268556:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204619.268557:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204619.268560:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6074 00000100:00080000:0.0:1788204619.268562:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204619.268565:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204619.268573:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204619.268586:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187979136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204619.268595:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187979008:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204619.268601:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204619.268608:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204619.268610:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204619.268615:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6074 00000100:00080000:0.0:1788204619.268619:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204619.268624:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204619.268629:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204619.268642:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187979264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204619.268652:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204619.268655:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204619.268657:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204619.268661:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6074 00000100:00080000:0.0:1788204619.268665:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788204619.268666:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204619.268674:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204619.268681:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204619.268693:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187979392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204619.268701:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204619.268703:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204619.268716:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204619.268719:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6074 00000100:00080000:0.0:1788204619.268722:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204619.268725:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204619.268731:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204619/real 1788204619] req@ffff8e828793d880 x1875062187979008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788204619.268732:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204619.268747:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187979520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204619.268753:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187979008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204619.268764:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204619.268767:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204619.268771:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187979008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204619.268800:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187979136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204619.268817:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204619.268825:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204619/real 1788204619] req@ffff8e828793fb80 x1875062187979136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204619.268833:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187979136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204619.268838:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204619.268840:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204619.268843:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187979136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204619.268846:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187979264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204619.268855:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204619.268860:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204619/real 1788204619] req@ffff8e828793d500 x1875062187979264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204619.268867:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187979264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204619.268872:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204619.268873:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204619.268875:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187979264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204619.268878:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187979392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204619.268888:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204619.268892:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204619/real 1788204619] req@ffff8e828793c380 x1875062187979392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204619.268900:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187979392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204619.268904:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204619.268905:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204619.268907:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187979392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204619.268910:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187979520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204619.268918:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204619.268923:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204619/real 1788204619] req@ffff8e828793e300 x1875062187979520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204619.268929:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187979520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204619.268934:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204619.268935:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204619.268937:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187979520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204619.972444:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204620.996478:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204622.020512:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204623.044510:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204624.068489:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204624.388449:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204624.388462:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204624.388467:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204624.388478:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6079 00000100:00080000:0.0:1788204624.388491:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204624.388500:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204624.388555:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187979648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204624.388586:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204624.388590:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204624.388592:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204624.388596:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6079 00000100:00080000:0.0:1788204624.388600:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204624.388604:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204624.388626:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204624.388640:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187979776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204624.388649:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187979648:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204624.388660:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204624.388664:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204624.388666:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204624.388670:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6079 00000100:00080000:0.0:1788204624.388675:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204624.388679:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204624.388684:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204624.388697:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187979904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204624.388706:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204624.388709:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204624.388711:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204624.388715:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6079 00000100:00080000:0.0:1788204624.388718:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788204624.388722:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204624.388727:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204624.388733:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204624.388743:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187980032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204624.388752:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204624.388755:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204624.388757:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204624.388761:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6079 00000100:00100000:3.0:1788204624.388768:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204624/real 1788204624] req@ffff8e828793f480 x1875062187979648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204624.388772:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204624.388780:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204624.388786:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204624.388789:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187979648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204624.388801:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187980160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204624.388804:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204624.388811:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204624.388817:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187979648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204624.388848:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187979776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204624.388872:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204624.388885:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204624/real 1788204624] req@ffff8e828793dc00 x1875062187979776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204624.388898:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187979776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204624.388906:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204624.388909:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204624.388912:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187979776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204624.388917:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187979904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204624.388936:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204624.388947:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204624/real 1788204624] req@ffff8e828793c000 x1875062187979904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204624.388954:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187979904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204624.388961:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204624.388964:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204624.388967:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187979904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204624.388973:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187980032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204624.388994:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204624.389005:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204624/real 1788204624] req@ffff8e828793ca80 x1875062187980032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204624.389013:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187980032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204624.389018:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204624.389020:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204624.389022:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187980032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204624.389025:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187980160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204624.389038:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204624.389043:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204624/real 1788204624] req@ffff8e828793f800 x1875062187980160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204624.389050:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187980160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204624.389054:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204624.389055:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204624.389057:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187980160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204625.092461:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204626.116501:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204627.140470:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204628.164516:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204629.188497:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204629.508476:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204629.508489:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204629.508495:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204629.508517:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6084 00000100:00080000:2.0:1788204629.508559:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204629.508568:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204629.508616:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187980288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204629.508652:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204629.508657:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204629.508659:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204629.508663:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6084 00000100:00080000:2.0:1788204629.508667:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204629.508671:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204629.508690:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204629.508709:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187980416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204629.508709:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187980288:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204629.508726:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204629.508729:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204629.508731:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204629.508735:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6084 00000100:00080000:2.0:1788204629.508740:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204629.508744:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204629.508750:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204629.508767:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187980544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204629.508777:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204629.508781:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204629.508783:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204629.508786:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6084 00000800:00000100:3.0:1788204629.508791:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204629.508792:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204629.508800:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204629.508806:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204629.508816:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187980672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204629.508823:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204629.508826:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204629.508827:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204629.508829:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6084 00000100:00080000:2.0:1788204629.508832:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204629.508836:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204629.508840:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204629.508858:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187980800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204629.508875:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204629/real 1788204629] req@ffff8e82894e4e00 x1875062187980288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204629.508896:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187980288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204629.508907:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204629.508912:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204629.508916:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187980288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204629.508941:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187980416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204629.508963:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204629.508974:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204629/real 1788204629] req@ffff8e82894e4a80 x1875062187980416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204629.508986:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187980416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204629.508993:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204629.508996:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204629.508999:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187980416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204629.509005:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187980544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204629.509024:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204629.509035:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204629/real 1788204629] req@ffff8e82894e6300 x1875062187980544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204629.509047:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187980544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204629.509061:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204629.509064:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204629.509067:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187980544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204629.509073:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187980672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204629.509092:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204629.509102:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204629/real 1788204629] req@ffff8e82894e7800 x1875062187980672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204629.509112:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187980672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204629.509120:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204629.509123:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204629.509126:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187980672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204629.509131:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187980800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204629.509147:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204629.509155:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204629/real 1788204629] req@ffff8e82894e6680 x1875062187980800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204629.509166:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187980800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204629.509172:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204629.509175:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204629.509179:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187980800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204630.212495:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204631.236486:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204632.260501:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204633.284458:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204634.308478:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204634.628434:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204634.628452:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204634.628456:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204634.628463:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6089 00000100:00080000:0.0:1788204634.628472:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204634.628478:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204634.628507:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187980928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204634.628526:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204634.628529:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204634.628530:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204634.628532:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6089 00000100:00080000:0.0:1788204634.628535:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204634.628537:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204634.628544:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204634.628551:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187981056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204634.628557:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204634.628559:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204634.628560:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204634.628562:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6089 00000100:00080000:0.0:1788204634.628564:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204634.628566:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204634.628569:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204634.628574:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187980928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204634.628577:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187981184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204634.628587:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204634.628588:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204634.628589:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204634.628591:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6089 00000100:00080000:0.0:1788204634.628593:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204634.628596:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204634.628598:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204634.628607:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187981312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204634.628616:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204634.628619:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204634.628620:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204634.628623:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6089 00000100:00080000:0.0:1788204634.628629:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204634.628632:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204634.628636:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204634.628647:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187981440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204634.628680:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187981056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204634.628720:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187981184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204634.628745:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187981312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204634.628764:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187981440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204635.332510:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204636.356502:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204637.380474:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788204637.700695:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788204637.700736:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788204637.700751:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204637.700790:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204637.700798:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204637.700804:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204637.700811:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:0.0:1788204637.701067:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204634/real 1788204637] req@ffff8e828793f100 x1875062187981184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204637.701133:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187981184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204637.701148:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204637.701154:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204637.701160:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187981184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204637.701178:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204634/real 1788204637] req@ffff8e828793ce00 x1875062187981312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204637.701192:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187981312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204637.701201:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204637.701205:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204637.701208:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187981312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204637.701216:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204634/real 1788204637] req@ffff8e828793f800 x1875062187981440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204637.701227:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187981440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204637.701235:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204637.701238:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204637.701242:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187981440:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204637.701251:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204634/real 1788204637] req@ffff8e828793e680 x1875062187980928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204637.701265:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187980928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204637.701272:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204637.701276:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204637.701279:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187980928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204637.701288:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204634/real 1788204637] req@ffff8e828793c700 x1875062187981056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204637.701306:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187981056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204637.701382:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204637.701402:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204637.701407:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187981056:192.168.201.121@tcp:38:kworker.0 00010000:00010000:1.0:1788204638.404631:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204639.428542:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204639.748437:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204639.748445:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204639.748450:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204639.748458:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6094 00000100:00080000:2.0:1788204639.748467:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204639.748478:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204639.748509:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187981568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204639.748529:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204639.748531:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204639.748533:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204639.748537:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6094 00000100:00080000:2.0:1788204639.748540:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204639.748543:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204639.748550:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204639.748559:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187981696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204639.748566:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204639.748568:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204639.748569:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204639.748576:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6094 00000100:00080000:2.0:1788204639.748580:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204639.748586:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204639.748589:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187981568:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788204639.748590:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204639.748605:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187981824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204639.748618:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204639.748620:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204639.748622:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204639.748625:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6094 00000100:00080000:2.0:1788204639.748627:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204639.748629:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204639.748633:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204639.748643:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187981952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204639.748649:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204639.748650:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204639.748652:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204639.748655:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6094 00000100:00080000:2.0:1788204639.748660:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204639.748664:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204639.748669:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204639.748683:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187982080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204639.748701:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204639.748766:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204639/real 1788204639] req@ffff8e82894e5880 x1875062187981568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204639.748789:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187981568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204639.748809:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204639.748815:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204639.748820:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187981568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204639.748830:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187981696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204639.748855:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204639.748869:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204639/real 1788204639] req@ffff8e82894e6a00 x1875062187981696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204639.748881:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187981696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204639.748890:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204639.748894:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204639.748897:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187981696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204639.748929:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187981824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204639.748949:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204639.748960:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204639/real 1788204639] req@ffff8e82894e7480 x1875062187981824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204639.748972:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187981824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204639.748979:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204639.748983:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204639.748986:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187981824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204639.748993:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187981952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204639.749014:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204639.749026:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204639/real 1788204639] req@ffff8e82894e4000 x1875062187981952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204639.749038:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187981952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204639.749046:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204639.749049:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204639.749052:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187981952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788204639.749060:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187982080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204639.749079:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204639.749090:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204639/real 1788204639] req@ffff8e82894e6680 x1875062187982080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204639.749101:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187982080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204639.749108:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204639.749112:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204639.749115:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187982080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204640.452545:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204641.476435:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204642.500431:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204643.524446:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204644.548463:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204644.868456:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204644.868467:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204644.868471:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204644.868481:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6099 00000100:00080000:0.0:1788204644.868493:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204644.868500:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204644.868542:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187982208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204644.868571:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204644.868574:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204644.868576:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204644.868579:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6099 00000100:00080000:0.0:1788204644.868581:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204644.868583:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204644.868592:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204644.868600:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187982336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204644.868606:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204644.868609:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204644.868610:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204644.868612:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6099 00000100:00080000:0.0:1788204644.868614:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204644.868616:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204644.868618:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204644.868626:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187982464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204644.868627:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187982208:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204644.868637:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204644.868640:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204644.868641:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204644.868643:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6099 00000100:00080000:0.0:1788204644.868646:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204644.868648:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204644.868651:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204644.868659:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187982592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204644.868664:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204644.868666:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204644.868667:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204644.868669:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6099 00000100:00080000:0.0:1788204644.868671:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204644.868672:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204644.868675:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204644.868681:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187982720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204644.868710:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204644.868763:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204644/real 1788204644] req@ffff8e828793c700 x1875062187982208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204644.868778:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187982208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204644.868787:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204644.868790:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204644.868793:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187982208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204644.868798:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187982336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204644.868815:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204644.868823:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204644/real 1788204644] req@ffff8e828793e680 x1875062187982336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204644.868830:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187982336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204644.868835:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204644.868837:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204644.868839:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187982336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204644.868859:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187982464:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204644.868869:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204644.868875:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204644/real 1788204644] req@ffff8e828793f800 x1875062187982464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204644.868886:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187982464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204644.868890:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204644.868892:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204644.868894:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187982464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204644.868897:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187982592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204644.868907:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204644.868912:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204644/real 1788204644] req@ffff8e828793ce00 x1875062187982592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204644.868919:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187982592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204644.868924:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204644.868925:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204644.868927:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187982592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788204644.868930:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187982720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204644.868939:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204644.868943:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204644/real 1788204644] req@ffff8e828793f100 x1875062187982720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204644.868950:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187982720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204644.868958:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204644.868960:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204644.868962:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187982720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204645.573407:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204646.596449:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204647.620537:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204648.644466:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204649.668481:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204649.988408:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204649.988418:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204649.988423:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204649.988434:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6105 00000100:00080000:0.0:1788204649.988445:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204649.988453:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204649.988485:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187982848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204649.988532:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204649.988536:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204649.988537:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204649.988543:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6105 00000100:00080000:0.0:1788204649.988547:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204649.988552:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204649.988560:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204649.988575:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187982976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204649.988586:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204649.988590:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187982848:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204649.988591:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204649.988595:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204649.988602:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6105 00000100:00080000:0.0:1788204649.988606:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204649.988611:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204649.988617:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204649.988632:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187983104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204649.988642:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204649.988645:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204649.988647:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204649.988651:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6105 00000100:00080000:0.0:1788204649.988655:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204649.988660:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204649.988665:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204649.988674:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187983232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204649.988681:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204649.988683:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204649.988685:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204649.988688:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6105 00000100:00080000:0.0:1788204649.988691:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204649.988694:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204649.988697:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204649.988702:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187983360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204649.988718:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204649.988896:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204649/real 1788204649] req@ffff8e828793ca80 x1875062187982848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204649.988924:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187982848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204649.988934:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204649.988938:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204649.988943:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187982848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204649.988965:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187982976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204649.988987:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204649.989000:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204649/real 1788204649] req@ffff8e828793c000 x1875062187982976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204649.989012:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187982976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204649.989020:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204649.989023:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204649.989026:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187982976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204649.989032:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187983104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204649.989050:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204649.989061:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204649/real 1788204649] req@ffff8e828793dc00 x1875062187983104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204649.989072:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187983104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204649.989080:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204649.989083:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204649.989086:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187983104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204649.989092:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187983232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204649.989110:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204649.989121:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204649/real 1788204649] req@ffff8e828793f480 x1875062187983232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204649.989131:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187983232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204649.989139:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204649.989142:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204649.989145:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187983232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204649.989150:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187983360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204649.989167:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204649.989196:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204649/real 1788204649] req@ffff8e828793e300 x1875062187983360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204649.989208:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187983360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204649.989215:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204649.989218:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204649.989221:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187983360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204650.692470:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204651.716449:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204652.740487:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204653.764451:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204654.788516:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204655.108402:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204655.108410:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204655.108414:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204655.108422:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6110 00000100:00080000:0.0:1788204655.108430:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204655.108437:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204655.108469:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187983488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204655.108489:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204655.108492:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204655.108494:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204655.108497:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6110 00000100:00080000:0.0:1788204655.108499:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204655.108502:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204655.108509:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204655.108517:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187983616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204655.108525:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204655.108528:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204655.108529:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204655.108531:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6110 00000100:00080000:0.0:1788204655.108533:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204655.108538:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187983488:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204655.108539:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204655.108545:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204655.108552:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187983744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204655.108559:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204655.108562:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204655.108563:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204655.108565:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6110 00000100:00080000:0.0:1788204655.108568:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204655.108572:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204655.108577:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204655.108588:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187983872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204655.108600:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788204655.108600:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204655.108607:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204655.108610:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204655.108616:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6110 00000100:00080000:0.0:1788204655.108621:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204655.108626:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204655.108631:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204655.108639:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204655/real 1788204655] req@ffff8e828793c380 x1875062187983488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204655.108645:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187984000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204655.108671:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187983488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204655.108682:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204655.108685:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204655.108690:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187983488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204655.108708:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187983616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204655.108723:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204655.108731:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204655/real 1788204655] req@ffff8e828793d500 x1875062187983616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204655.108739:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187983616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204655.108745:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204655.108747:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204655.108749:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187983616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204655.108753:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187983744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204655.108763:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204655.108769:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204655/real 1788204655] req@ffff8e828793fb80 x1875062187983744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204655.108776:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187983744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204655.108781:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204655.108783:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204655.108785:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187983744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204655.108789:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187983872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204655.108799:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204655.108805:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204655/real 1788204655] req@ffff8e828793d880 x1875062187983872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204655.108812:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187983872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204655.108817:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204655.108819:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204655.108821:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187983872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204655.108824:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187984000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204655.108833:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204655.108838:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204655/real 1788204655] req@ffff8e828793e300 x1875062187984000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204655.108845:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187984000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204655.108851:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204655.108854:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204655.108857:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187984000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204655.812494:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204656.836455:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204657.860530:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204658.884454:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204659.908499:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788204660.228442:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204660.228450:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788204660.228454:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204660.228463:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6115 00000100:00080000:3.0:1788204660.228472:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204660.228478:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204660.228508:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062187984128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204660.228543:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204660.228546:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788204660.228547:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204660.228550:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6115 00000100:00080000:3.0:1788204660.228552:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204660.228555:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204660.228561:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204660.228570:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062187984256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204660.228577:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204660.228580:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788204660.228581:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204660.228584:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6115 00000100:00080000:3.0:1788204660.228587:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204660.228589:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204660.228593:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204660.228596:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187984128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204660.228602:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062187984384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204660.228615:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204660.228617:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788204660.228619:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204660.228621:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6115 00000100:00080000:3.0:1788204660.228624:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204660.228626:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204660.228629:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204660.228638:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062187984512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204660.228645:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788204660.228647:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788204660.228648:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204660.228651:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6115 00000100:00080000:3.0:1788204660.228653:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788204660.228657:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788204660.228661:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788204660.228666:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204660.228674:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062187984640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204660.228697:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204660/real 1788204660] req@ffff8e8289538380 x1875062187984128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204660.228710:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062187984128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204660.228718:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204660.228721:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204660.228723:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187984128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204660.228728:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187984256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204660.228742:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204660.228750:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204660/real 1788204660] req@ffff8e8289539f80 x1875062187984256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204660.228758:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062187984256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204660.228763:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204660.228765:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204660.228767:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187984256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204660.228785:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187984384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204660.228795:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204660.228802:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204660/real 1788204660] req@ffff8e828953a680 x1875062187984384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204660.228809:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062187984384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204660.228813:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204660.228815:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204660.228817:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187984384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204660.228820:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187984512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204660.228831:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204660.228836:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204660/real 1788204660] req@ffff8e8289539500 x1875062187984512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204660.228843:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062187984512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204660.228849:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204660.228850:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204660.228853:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187984512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788204660.228856:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187984640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204660.228865:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204660.228870:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204660/real 1788204660] req@ffff8e8289539880 x1875062187984640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204660.228877:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062187984640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204660.228881:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204660.228883:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204660.228885:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187984640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204660.932495:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204661.956473:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204662.980490:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204664.004498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204665.028495:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204665.348463:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204665.348475:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204665.348489:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204665.348499:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6120 00000100:00080000:2.0:1788204665.348511:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204665.348519:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204665.348557:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187984768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204665.348589:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204665.348593:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204665.348595:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204665.348600:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6120 00000100:00080000:2.0:1788204665.348604:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204665.348609:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204665.348618:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204665.348635:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187984896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204665.348637:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187984768:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204665.348662:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204665.348666:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204665.348669:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204665.348674:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6120 00000100:00080000:2.0:1788204665.348678:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204665.348684:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204665.348689:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204665.348701:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187985024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204665.348709:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204665.348718:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204665.348722:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204665.348725:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204665.348731:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6120 00000100:00080000:2.0:1788204665.348735:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204665.348740:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204665.348745:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204665.348752:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204665/real 1788204665] req@ffff8e82894e7800 x1875062187984768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204665.348760:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187985152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204665.348774:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187984768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204665.348775:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204665.348786:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204665.348790:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204665.348801:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6120 00000100:00080000:1.0:1788204665.348804:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204665.348808:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204665.348811:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:2.0:1788204665.348817:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204665.348818:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187984768:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788204665.348828:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204665.348843:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187985280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204665.348846:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187984896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204665.348881:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204665.348897:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204665/real 1788204665] req@ffff8e82894e6300 x1875062187984896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204665.348925:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187984896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204665.348933:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204665.348936:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204665.348938:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187984896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204665.348943:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187985024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204665.348959:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204665.348967:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204665/real 1788204665] req@ffff8e82894e4a80 x1875062187985024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204665.348976:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187985024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204665.348981:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204665.348983:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204665.348985:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187985024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204665.348989:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187985152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204665.349000:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204665.349006:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204665/real 1788204665] req@ffff8e82894e4e00 x1875062187985152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204665.349013:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187985152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204665.349018:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204665.349020:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204665.349022:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187985152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788204665.349032:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187985280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204665.349042:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204665.349048:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204665/real 1788204665] req@ffff8e82894e6d80 x1875062187985280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204665.349055:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187985280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204665.349060:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204665.349061:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204665.349063:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187985280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204666.052529:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204667.076549:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204668.100552:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204669.124491:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204670.148465:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204670.468413:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204670.468421:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204670.468425:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204670.468434:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6125 00000100:00080000:2.0:1788204670.468443:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204670.468450:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204670.468482:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187985408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204670.468508:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204670.468511:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204670.468513:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204670.468517:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6125 00000100:00080000:2.0:1788204670.468519:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204670.468523:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204670.468530:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204670.468540:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187985536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204670.468547:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187985408:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204670.468548:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204670.468555:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204670.468558:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204670.468561:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6125 00000100:00080000:2.0:1788204670.468563:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204670.468565:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204670.468569:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204670.468575:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187985664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204670.468581:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204670.468583:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204670.468585:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204670.468587:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6125 00000100:00080000:2.0:1788204670.468589:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204670.468592:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204670.468595:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204670.468603:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187985792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204670.468604:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204670.468611:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204670.468614:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204670.468615:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204670.468618:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6125 00000100:00080000:2.0:1788204670.468620:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204670.468622:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204670.468625:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204670.468632:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187985920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204670.468652:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204670/real 1788204670] req@ffff8e82894e7100 x1875062187985408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204670.468670:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187985408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204670.468678:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204670.468680:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204670.468683:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187985408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204670.468700:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187985536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204670.468716:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204670.468724:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204670/real 1788204670] req@ffff8e82894e5c00 x1875062187985536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204670.468732:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187985536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204670.468736:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204670.468738:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204670.468740:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187985536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204670.468743:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187985664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204670.468755:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204670.468764:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204670/real 1788204670] req@ffff8e82894e4380 x1875062187985664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204670.468781:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187985664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204670.468788:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204670.468791:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204670.468794:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187985664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204670.468799:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187985792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204670.468831:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204670.468843:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204670/real 1788204670] req@ffff8e82894e5f80 x1875062187985792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204670.468855:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187985792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204670.468862:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204670.468865:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204670.468868:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187985792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788204670.468874:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187985920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204670.468892:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204670.468902:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204670/real 1788204670] req@ffff8e82894e6d80 x1875062187985920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204670.468913:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187985920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204670.468926:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204670.468930:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204670.468933:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187985920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204671.172462:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204672.196500:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204673.220471:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204674.244487:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204675.268505:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204675.588422:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204675.588430:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204675.588434:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204675.588442:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6130 00000100:00080000:0.0:1788204675.588450:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204675.588456:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204675.588496:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187986048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204675.588518:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204675.588521:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204675.588522:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204675.588526:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6130 00000100:00080000:0.0:1788204675.588529:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204675.588532:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204675.588540:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204675.588551:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187986176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204675.588551:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187986048:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204675.588564:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204675.588567:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204675.588568:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204675.588570:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6130 00000100:00080000:0.0:1788204675.588572:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204675.588574:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204675.588577:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204675.588585:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187986304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204675.588594:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204675.588597:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204675.588598:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204675.588600:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6130 00000100:00080000:0.0:1788204675.588602:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204675.588604:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204675.588606:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788204675.588608:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204675.588616:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187986432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204675.588623:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204675.588625:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204675.588627:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204675.588630:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6130 00000100:00080000:0.0:1788204675.588632:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204675.588633:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204675.588636:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204675.588641:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187986560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204675.588662:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204675/real 1788204675] req@ffff8e828793f480 x1875062187986048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204675.588676:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187986048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204675.588683:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204675.588685:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204675.588688:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187986048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204675.588706:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187986176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204675.588727:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204675.588748:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204675/real 1788204675] req@ffff8e828793dc00 x1875062187986176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204675.588756:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187986176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204675.588761:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204675.588763:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204675.588765:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187986176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204675.588768:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187986304:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204675.588780:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204675.588786:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204675/real 1788204675] req@ffff8e828793c000 x1875062187986304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204675.588792:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187986304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204675.588797:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204675.588798:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204675.588800:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187986304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204675.588803:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187986432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204675.588813:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204675.588818:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204675/real 1788204675] req@ffff8e828793ca80 x1875062187986432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204675.588824:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187986432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204675.588829:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204675.588830:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204675.588832:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187986432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204675.588834:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187986560:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204675.588842:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204675.588847:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204675/real 1788204675] req@ffff8e828793f100 x1875062187986560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204675.588853:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187986560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204675.588857:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204675.588858:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204675.588860:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187986560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204676.292466:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204677.316476:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204678.340440:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204679.364561:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204680.388494:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204680.708435:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204680.708444:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204680.708449:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204680.708458:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6135 00000100:00080000:0.0:1788204680.708468:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204680.708474:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204680.708509:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187986688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204680.708537:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204680.708540:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204680.708541:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204680.708544:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6135 00000100:00080000:0.0:1788204680.708547:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204680.708549:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204680.708557:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204680.708566:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187986816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204680.708573:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204680.708575:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204680.708578:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788204680.708579:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187986688:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204680.708582:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6135 00000100:00080000:0.0:1788204680.708587:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204680.708589:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204680.708592:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204680.708599:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187986944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204680.708609:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204680.708611:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204680.708613:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204680.708615:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6135 00000100:00080000:0.0:1788204680.708617:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204680.708619:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204680.708621:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204680.708628:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187987072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788204680.708634:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204680.708664:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204680.708669:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204680.708671:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204680.708675:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6135 00000100:00080000:0.0:1788204680.708679:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204680.708683:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204680.708688:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204680.708691:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204680/real 1788204680] req@ffff8e828793ce00 x1875062187986688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204680.708703:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187987200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204680.708712:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187986688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204680.708730:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204680.708734:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204680.708739:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187986688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204680.708760:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187986816:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204680.708783:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204680.708793:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204680/real 1788204680] req@ffff8e828793f800 x1875062187986816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204680.708801:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187986816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204680.708806:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204680.708808:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204680.708810:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187986816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204680.708813:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187986944:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204680.708824:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204680.708830:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204680/real 1788204680] req@ffff8e828793e680 x1875062187986944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204680.708836:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187986944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204680.708841:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204680.708842:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204680.708844:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187986944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204680.708849:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187987072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204680.708865:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204680.708875:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204680/real 1788204680] req@ffff8e828793c700 x1875062187987072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204680.708886:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187987072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204680.708894:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204680.708896:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204680.708898:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187987072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204680.708902:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187987200:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204680.708915:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204680.708921:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204680/real 1788204680] req@ffff8e828793f100 x1875062187987200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204680.708928:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187987200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204680.708932:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204680.708934:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204680.708935:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187987200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204681.412468:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204682.436504:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204683.460463:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204684.484495:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204685.508497:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204685.828622:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204685.828630:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204685.828635:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204685.828645:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6140 00000100:00080000:1.0:1788204685.828656:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204685.828662:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204685.828698:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187987328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204685.828716:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204685.828719:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204685.828720:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204685.828723:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6140 00000100:00080000:1.0:1788204685.828725:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204685.828728:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204685.828735:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204685.828746:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187987456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204685.828754:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204685.828755:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187987328:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204685.828756:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204685.828759:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204685.828765:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6140 00000100:00080000:1.0:1788204685.828767:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204685.828769:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204685.828772:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204685.828780:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187987584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204685.828786:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204685.828788:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204685.828789:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204685.828791:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6140 00000100:00080000:1.0:1788204685.828794:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204685.828796:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204685.828800:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204685.828806:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187987712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204685.828811:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204685.828813:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204685.828814:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204685.828817:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6140 00000100:00080000:1.0:1788204685.828826:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204685.828828:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204685.828831:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788204685.828838:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204685.828839:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187987840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204685.828944:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204685/real 1788204685] req@ffff8e828d720e00 x1875062187987328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204685.829086:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187987328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204685.829137:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204685.829143:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204685.829148:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187987328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204685.829179:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187987456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204685.829205:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204685.829219:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204685/real 1788204685] req@ffff8e828d722680 x1875062187987456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204685.829232:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187987456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204685.829240:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204685.829244:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204685.829247:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187987456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204685.829253:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187987584:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204685.829273:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204685.829284:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204685/real 1788204685] req@ffff8e828d720000 x1875062187987584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204685.829296:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187987584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204685.829331:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204685.829336:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204685.829340:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187987584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204685.829347:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187987712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204685.829370:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204685.829382:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204685/real 1788204685] req@ffff8e828d721500 x1875062187987712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204685.829394:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187987712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204685.829402:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204685.829405:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204685.829408:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187987712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204685.829414:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187987840:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204685.829433:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204685.829444:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204685/real 1788204685] req@ffff8e828d723800 x1875062187987840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204685.829455:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187987840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204685.829463:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204685.829466:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204685.829469:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187987840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204686.532523:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204687.556564:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204688.580444:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204689.604538:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204690.628506:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204690.948377:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204690.948391:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204690.948397:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204690.948409:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6146 00000100:00080000:1.0:1788204690.948422:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204690.948431:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204690.948492:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187987968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204690.948519:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204690.948523:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204690.948525:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204690.948530:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6146 00000100:00080000:1.0:1788204690.948535:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204690.948540:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204690.948550:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204690.948563:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187987968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204690.948564:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187988096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204690.948578:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204690.948581:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204690.948582:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204690.948585:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6146 00000100:00080000:1.0:1788204690.948587:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204690.948590:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204690.948593:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204690.948600:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187988224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204690.948607:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204690.948609:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204690.948610:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204690.948612:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6146 00000100:00080000:1.0:1788204690.948614:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204690.948617:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204690.948620:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204690.948626:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187988352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204690.948632:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204690.948633:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204690.948634:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204690.948637:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6146 00000100:00080000:1.0:1788204690.948640:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204690.948652:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:3.0:1788204690.948652:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:1.0:1788204690.948660:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204690.948672:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187988480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204690.948706:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204690/real 1788204690] req@ffff8e828d722300 x1875062187987968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204690.948725:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187987968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204690.948735:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204690.948739:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204690.948743:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187987968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204690.948770:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187988096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204690.948800:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204690.948812:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204690/real 1788204690] req@ffff8e828d720380 x1875062187988096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204690.948820:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187988096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204690.948825:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204690.948828:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204690.948830:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187988096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204690.948833:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187988224:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204690.948845:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204690.948852:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204690/real 1788204690] req@ffff8e828d722d80 x1875062187988224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204690.948859:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187988224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204690.948864:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204690.948866:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204690.948868:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187988224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204690.948873:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187988352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204690.948892:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204690.948902:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204690/real 1788204690] req@ffff8e828d721180 x1875062187988352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204690.948913:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187988352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204690.948921:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204690.948924:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204690.948927:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187988352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204690.948932:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187988480:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204690.948951:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204690.948961:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204690/real 1788204690] req@ffff8e828d720a80 x1875062187988480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204690.948971:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187988480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204690.948980:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204690.948983:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204690.948986:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187988480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204691.652497:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204692.676526:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204693.700507:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204694.724503:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204695.748447:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204696.068413:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204696.068422:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204696.068427:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204696.068436:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6151 00000100:00080000:1.0:1788204696.068446:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204696.068453:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204696.068484:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187988608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204696.068507:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204696.068509:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204696.068510:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204696.068513:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6151 00000100:00080000:1.0:1788204696.068516:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204696.068519:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204696.068526:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204696.068535:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187988736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204696.068539:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187988608:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204696.068543:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204696.068548:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204696.068549:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204696.068552:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6151 00000100:00080000:1.0:1788204696.068554:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204696.068556:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204696.068558:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204696.068564:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187988864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204696.068569:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204696.068571:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204696.068572:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204696.068575:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6151 00000100:00080000:1.0:1788204696.068577:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204696.068579:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204696.068582:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204696.068587:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187988992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204696.068592:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204696.068595:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204696.068596:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204696.068598:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6151 00000100:00080000:1.0:1788204696.068601:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204696.068605:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:3.0:1788204696.068606:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:1.0:1788204696.068609:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204696.068617:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187989120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204696.068642:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204696/real 1788204696] req@ffff8e828d720a80 x1875062187988608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204696.068655:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187988608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204696.068662:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204696.068664:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204696.068667:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187988608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204696.068684:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187988736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204696.068698:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204696.068705:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204696/real 1788204696] req@ffff8e828d721180 x1875062187988736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204696.068712:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187988736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204696.068717:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204696.068719:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204696.068721:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187988736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204696.068725:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187988864:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204696.068734:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204696.068739:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204696/real 1788204696] req@ffff8e828d722d80 x1875062187988864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204696.068745:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187988864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204696.068750:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204696.068751:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204696.068753:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187988864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204696.068755:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187988992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204696.068764:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204696.068769:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204696/real 1788204696] req@ffff8e828d720380 x1875062187988992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204696.068775:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187988992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204696.068782:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204696.068785:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204696.068788:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187988992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204696.068793:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187989120:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204696.068811:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204696.068821:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204696/real 1788204696] req@ffff8e828d722300 x1875062187989120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204696.068832:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187989120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204696.068839:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204696.068842:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204696.068845:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187989120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204696.772471:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204697.796479:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204698.820512:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204699.844493:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204700.868498:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204701.188585:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204701.188599:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204701.188605:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204701.188619:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6156 00000100:00080000:2.0:1788204701.188633:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204701.188643:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204701.188689:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187989248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204701.188718:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204701.188722:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204701.188723:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204701.188729:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6156 00000100:00080000:2.0:1788204701.188733:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204701.188738:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204701.188748:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204701.188760:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187989248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204701.188764:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187989376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204701.188784:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204701.188788:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204701.188789:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204701.188793:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6156 00000100:00080000:2.0:1788204701.188797:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204701.188801:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204701.188806:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204701.188817:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187989504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204701.188833:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204701.188836:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204701.188839:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204701.188842:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6156 00000100:00080000:2.0:1788204701.188853:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204701.188861:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204701.188866:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204701.188881:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187989632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204701.188888:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187989376:192.168.201.121@tcp:38:kworker.0 00000100:00080000:2.0:1788204701.188896:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204701.188901:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204701.188903:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204701.188907:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6156 00000100:00080000:2.0:1788204701.188915:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204701.188920:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204701.188924:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187989504:192.168.201.121@tcp:38:kworker.0 00000008:00100000:2.0:1788204701.188925:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204701.188942:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187989760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204701.188955:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187989632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204701.188977:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187989760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204701.892532:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204702.916493:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204703.940576:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788204704.260585:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:1.0:1788204704.260628:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788204704.260644:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788204704.260688:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788204704.260697:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788204704.260705:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788204704.260711:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:3.0:1788204704.260792:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204701/real 1788204704] req@ffff8e82894e7800 x1875062187989632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204704.260815:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187989632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204704.260825:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204704.260830:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204704.260834:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187989632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204704.260841:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204701/real 1788204704] req@ffff8e82894e6680 x1875062187989760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204704.260861:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187989760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204704.260868:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204704.260871:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204704.260874:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187989760:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204704.260879:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204701/real 1788204704] req@ffff8e82894e4a80 x1875062187989376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204704.260891:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187989376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204704.260897:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204704.260900:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204704.260904:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187989376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204704.260910:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204701/real 1788204704] req@ffff8e82894e6300 x1875062187989504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204704.260921:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187989504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204704.260928:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204704.260931:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204704.260934:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187989504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204704.260940:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204701/real 1788204704] req@ffff8e82894e4e00 x1875062187989248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204704.260951:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187989248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204704.260958:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204704.260961:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204704.260964:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187989248:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788204704.964522:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204705.988532:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204706.308426:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204706.308439:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204706.308444:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204706.308458:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6161 00000100:00080000:2.0:1788204706.308473:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204706.308482:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204706.308527:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062187989888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204706.308558:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204706.308562:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204706.308564:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204706.308569:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6161 00000100:00080000:2.0:1788204706.308573:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204706.308579:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204706.308590:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204706.308599:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187989888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204706.308608:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062187990016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204706.308620:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204706.308623:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204706.308625:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204706.308629:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6161 00000100:00080000:2.0:1788204706.308633:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204706.308637:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204706.308643:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204706.308655:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062187990144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204706.308664:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204706.308669:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204706.308671:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204706.308675:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6161 00000100:00080000:2.0:1788204706.308679:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204706.308685:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:3.0:1788204706.308687:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:2.0:1788204706.308694:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204706.308709:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062187990272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204706.308718:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204706.308722:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204706.308723:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204706.308728:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6161 00000100:00080000:2.0:1788204706.308732:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204706.308736:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204706.308741:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204706.308746:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204706/real 1788204706] req@ffff8e82894e4000 x1875062187989888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204706.308758:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062187990400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204706.308771:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062187989888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204706.308785:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204706.308790:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204706.308794:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187989888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204706.308820:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187990016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204706.308853:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204706.308865:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204706/real 1788204706] req@ffff8e82894e7480 x1875062187990016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204706.308878:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062187990016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204706.308884:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204706.308888:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204706.308891:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187990016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204706.308898:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187990144:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204706.308916:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204706.308928:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204706/real 1788204706] req@ffff8e82894e6a00 x1875062187990144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204706.308939:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062187990144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204706.308946:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204706.308949:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204706.308951:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187990144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204706.308957:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187990272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204706.308975:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204706.308984:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204706/real 1788204706] req@ffff8e82894e5880 x1875062187990272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204706.308994:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062187990272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204706.309000:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204706.309004:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204706.309006:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187990272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204706.309012:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187990400:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204706.309030:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204706.309040:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204706/real 1788204706] req@ffff8e82894e4e00 x1875062187990400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204706.309056:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062187990400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204706.309064:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204706.309067:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204706.309070:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187990400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204707.012487:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204708.036487:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204709.060518:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204710.084432:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204711.108419:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204711.428435:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204711.428444:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204711.428448:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204711.428458:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6166 00000100:00080000:0.0:1788204711.428468:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204711.428475:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204711.428508:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187990528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204711.428533:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204711.428537:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204711.428539:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204711.428544:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6166 00000100:00080000:0.0:1788204711.428548:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204711.428552:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204711.428562:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204711.428573:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187990528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204711.428578:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187990656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204711.428590:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204711.428594:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204711.428596:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204711.428600:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6166 00000100:00080000:0.0:1788204711.428603:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204711.428608:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204711.428613:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788204711.428626:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204711.428628:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187990784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204711.428638:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204711.428642:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204711.428644:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204711.428648:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6166 00000100:00080000:0.0:1788204711.428652:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204711.428656:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204711.428661:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204711.428670:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204711/real 1788204711] req@ffff8e828793ca80 x1875062187990528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204711.428674:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187990912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204711.428696:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204711.428700:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204711.428702:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788204711.428706:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187990528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204711.428717:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6166 00000100:00080000:3.0:1788204711.428719:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204711.428726:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204711.428726:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204711.428733:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187990528:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204711.428735:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204711.428742:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204711.428754:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187991040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204711.428760:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187990656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204711.428789:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204711.428803:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204711/real 1788204711] req@ffff8e828793c000 x1875062187990656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204711.428819:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187990656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204711.428824:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204711.428825:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204711.428828:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187990656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204711.428831:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187990784:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204711.428855:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204711.428862:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204711/real 1788204711] req@ffff8e828793dc00 x1875062187990784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204711.428869:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187990784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204711.428873:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204711.428875:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204711.428877:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187990784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204711.428880:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187990912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204711.428890:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204711.428895:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204711/real 1788204711] req@ffff8e828793f480 x1875062187990912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204711.428901:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187990912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204711.428909:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204711.428910:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204711.428912:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187990912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204711.428921:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187991040:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204711.428930:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204711.428935:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204711/real 1788204711] req@ffff8e828793e300 x1875062187991040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204711.428941:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187991040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204711.428945:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204711.428947:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204711.428948:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187991040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204712.132432:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204713.156482:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204714.180462:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204715.204506:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204716.228518:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204716.548463:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204716.548472:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204716.548475:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204716.548483:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6171 00000100:00080000:0.0:1788204716.548491:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204716.548495:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204716.548531:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187991168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204716.548553:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204716.548556:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204716.548558:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204716.548563:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6171 00000100:00080000:0.0:1788204716.548567:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204716.548571:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204716.548579:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204716.548595:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187991296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204716.548595:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187991168:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204716.548624:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204716.548626:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204716.548628:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204716.548631:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6171 00000100:00080000:0.0:1788204716.548634:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204716.548637:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204716.548647:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204716.548658:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187991424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204716.548665:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204716.548668:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204716.548669:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204716.548672:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6171 00000100:00080000:0.0:1788204716.548675:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788204716.548682:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204716.548683:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204716.548689:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204716.548698:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187991552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204716.548705:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204716.548708:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204716.548710:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204716.548712:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6171 00000100:00080000:0.0:1788204716.548715:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204716.548717:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204716.548722:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204716.548737:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204716/real 1788204716] req@ffff8e828793d880 x1875062187991168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204716.548738:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187991680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204716.548771:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187991168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204716.548781:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204716.548785:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204716.548788:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187991168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204716.548813:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187991296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204716.548831:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204716.548843:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204716/real 1788204716] req@ffff8e828793fb80 x1875062187991296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204716.548857:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187991296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204716.548866:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204716.548870:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204716.548874:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187991296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204716.548882:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187991424:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204716.548899:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204716.548907:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204716/real 1788204716] req@ffff8e828793d500 x1875062187991424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204716.548915:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187991424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204716.548921:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204716.548923:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204716.548925:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187991424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204716.548929:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187991552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204716.548940:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204716.548946:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204716/real 1788204716] req@ffff8e828793c380 x1875062187991552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204716.548955:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187991552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204716.548961:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204716.548964:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204716.548967:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187991552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204716.548973:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187991680:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204716.548994:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204716.549007:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204716/real 1788204716] req@ffff8e828793e300 x1875062187991680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204716.549019:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187991680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204716.549042:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204716.549045:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204716.549047:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187991680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204717.252422:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204718.276472:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204719.300487:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204720.324435:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204721.348543:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204721.668503:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204721.668512:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204721.668518:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204721.668532:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6176 00000100:00080000:0.0:1788204721.668545:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204721.668555:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204721.668602:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187991808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204721.668644:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204721.668649:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204721.668651:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204721.668658:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6176 00000100:00080000:0.0:1788204721.668663:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204721.668669:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204721.668682:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204721.668700:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187991808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204721.668702:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187991936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204721.668725:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204721.668728:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204721.668730:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204721.668734:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6176 00000100:00080000:0.0:1788204721.668738:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204721.668743:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204721.668749:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204721.668762:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187992064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204721.668772:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204721.668776:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204721.668778:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204721.668782:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6176 00000100:00080000:0.0:1788204721.668788:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788204721.668788:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204721.668796:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204721.668802:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204721.668814:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187992192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204721.668823:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204721.668826:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204721.668828:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204721.668832:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6176 00000100:00080000:0.0:1788204721.668835:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204721.668840:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204721.668853:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204721.668869:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187992320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204721.669034:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204721/real 1788204721] req@ffff8e828793f480 x1875062187991808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204721.669053:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187991808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204721.669068:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204721.669109:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204721.669114:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187991808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204721.669145:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187991936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204721.669166:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204721.669177:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204721/real 1788204721] req@ffff8e828793dc00 x1875062187991936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204721.669186:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187991936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204721.669191:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204721.669194:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204721.669196:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187991936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204721.669200:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187992064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204721.669211:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204721.669218:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204721/real 1788204721] req@ffff8e828793c000 x1875062187992064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204721.669225:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187992064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204721.669230:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204721.669232:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204721.669234:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187992064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204721.669238:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187992192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204721.669251:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204721.669261:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204721/real 1788204721] req@ffff8e828793ca80 x1875062187992192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204721.669274:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187992192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204721.669294:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204721.669298:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204721.669302:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187992192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204721.669333:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187992320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204721.669353:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204721.669362:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204721/real 1788204721] req@ffff8e828793f100 x1875062187992320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204721.669370:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187992320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204721.669376:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204721.669379:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204721.669382:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187992320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204722.372525:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204723.396493:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204724.420479:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204725.444492:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204726.468548:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204726.788470:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204726.788485:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204726.788491:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204726.788505:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6181 00000100:00080000:0.0:1788204726.788517:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204726.788526:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204726.788575:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187992448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204726.788606:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204726.788610:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204726.788612:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204726.788617:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6181 00000100:00080000:0.0:1788204726.788621:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204726.788638:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187992448:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204726.788640:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204726.788655:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204726.788671:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187992576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204726.788681:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204726.788686:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204726.788688:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204726.788691:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6181 00000100:00080000:0.0:1788204726.788695:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204726.788699:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204726.788704:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204726.788714:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187992704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204726.788721:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204726.788728:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204726.788732:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204726.788734:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204726.788738:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6181 00000100:00080000:0.0:1788204726.788754:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204726.788759:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204726.788763:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204726.788775:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204726/real 1788204726] req@ffff8e828793c700 x1875062187992448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204726.788777:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187992832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204726.788805:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204726.788809:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204726.788811:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204726.788815:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6181 00000100:00080000:0.0:1788204726.788819:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204726.788820:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187992448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204726.788826:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204726.788834:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788204726.788841:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204726.788859:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204726.788866:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187992448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204726.788868:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187992960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204726.788897:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187992576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204726.788916:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204726.788926:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204726/real 1788204726] req@ffff8e828793e680 x1875062187992576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204726.788934:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187992576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204726.788940:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204726.788942:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204726.788944:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187992576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204726.788947:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187992704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204726.788961:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204726.788972:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204726/real 1788204726] req@ffff8e828793f800 x1875062187992704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204726.788990:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187992704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204726.789004:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204726.789006:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204726.789009:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187992704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204726.789012:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187992832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204726.789035:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204726.789042:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204726/real 1788204726] req@ffff8e828793ce00 x1875062187992832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204726.789049:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187992832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204726.789054:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204726.789056:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204726.789057:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187992832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204726.789060:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187992960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204726.789070:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204726.789079:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204726/real 1788204726] req@ffff8e828793f100 x1875062187992960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204726.789085:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187992960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204726.789090:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204726.789091:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204726.789094:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187992960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204727.492504:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204728.516482:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204729.540486:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204730.564486:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204731.588487:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204731.908420:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204731.908428:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204731.908432:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204731.908440:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6186 00000100:00080000:1.0:1788204731.908450:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204731.908455:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204731.908484:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187993088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204731.908511:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204731.908514:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204731.908516:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204731.908520:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6186 00000100:00080000:1.0:1788204731.908523:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204731.908527:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204731.908537:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204731.908549:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187993216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204731.908553:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187993088:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204731.908558:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204731.908562:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204731.908563:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204731.908566:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6186 00000100:00080000:1.0:1788204731.908568:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204731.908571:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204731.908576:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204731.908587:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187993344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204731.908597:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204731.908600:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204731.908601:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204731.908605:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6186 00000100:00080000:1.0:1788204731.908608:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204731.908611:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204731.908615:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204731.908624:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187993472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204731.908633:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788204731.908634:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204731.908639:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204731.908641:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204731.908646:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6186 00000100:00080000:1.0:1788204731.908650:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204731.908654:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204731.908658:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204731.908669:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187993600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204731.908680:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204731/real 1788204731] req@ffff8e828d723800 x1875062187993088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204731.908698:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187993088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204731.908708:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204731.908712:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204731.908716:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187993088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204731.908739:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187993216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204731.908790:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204731.908803:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204731/real 1788204731] req@ffff8e828d721500 x1875062187993216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204731.908815:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187993216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204731.908823:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204731.908831:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204731.908834:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187993216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204731.908840:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187993344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204731.908859:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204731.908871:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204731/real 1788204731] req@ffff8e828d720000 x1875062187993344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204731.908882:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187993344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204731.908890:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204731.908892:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204731.908894:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187993344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204731.908898:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187993472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204731.908912:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204731.908919:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204731/real 1788204731] req@ffff8e828d722680 x1875062187993472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204731.908925:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187993472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204731.908929:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204731.908931:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204731.908932:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187993472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204731.908935:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187993600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204731.908944:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204731.908948:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204731/real 1788204731] req@ffff8e828d720e00 x1875062187993600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204731.908954:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187993600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204731.908959:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204731.908961:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204731.908962:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187993600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204732.612486:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204733.636496:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204734.660468:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204735.684479:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204736.708433:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204737.028438:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204737.028447:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204737.028451:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204737.028459:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6192 00000100:00080000:0.0:1788204737.028467:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204737.028473:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204737.028506:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187993728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204737.028528:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204737.028531:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204737.028532:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204737.028535:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6192 00000100:00080000:0.0:1788204737.028538:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204737.028540:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204737.028558:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204737.028566:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187993856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204737.028572:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204737.028576:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204737.028577:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204737.028580:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6192 00000100:00080000:0.0:1788204737.028582:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204737.028584:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204737.028587:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204737.028593:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187993984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204737.028599:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204737.028601:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204737.028603:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204737.028606:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6192 00000100:00080000:0.0:1788204737.028608:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204737.028612:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204737.028615:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187993728:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788204737.028616:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204737.028628:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187994112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204737.028635:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204737.028649:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204737.028650:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204737.028652:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6192 00000100:00080000:0.0:1788204737.028654:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204737.028657:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204737.028660:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204737.028683:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187994240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204737.028720:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204737.028756:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204737/real 1788204737] req@ffff8e828793ca80 x1875062187993728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204737.028769:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187993728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204737.028777:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204737.028780:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204737.028783:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187993728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204737.028788:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187993856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204737.028802:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204737.028810:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204737/real 1788204737] req@ffff8e828793c000 x1875062187993856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204737.028818:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187993856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204737.028823:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204737.028824:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204737.028826:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187993856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204737.028830:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187993984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204737.028840:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204737.028846:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204737/real 1788204737] req@ffff8e828793dc00 x1875062187993984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204737.028853:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187993984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204737.028858:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204737.028860:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204737.028862:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187993984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204737.028880:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187994112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204737.028890:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204737.028896:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204737/real 1788204737] req@ffff8e828793f480 x1875062187994112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204737.028903:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187994112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204737.028907:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204737.028909:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204737.028911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187994112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204737.028914:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187994240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204737.028923:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204737.028928:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204737/real 1788204737] req@ffff8e828793e300 x1875062187994240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204737.028936:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187994240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204737.028941:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204737.028942:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204737.028944:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187994240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204737.732471:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204738.756468:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204739.780481:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204740.804514:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204741.828489:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204742.148484:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204742.148495:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204742.148500:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204742.148512:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6197 00000100:00080000:2.0:1788204742.148525:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204742.148532:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204742.148573:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062187994368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204742.148640:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204742.148644:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204742.148646:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204742.148650:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6197 00000100:00080000:2.0:1788204742.148653:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204742.148656:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204742.148667:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204742.148680:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062187994496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204742.148689:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204742.148691:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204742.148693:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204742.148699:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6197 00000100:00080000:2.0:1788204742.148703:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204742.148703:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187994368:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204742.148707:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204742.148716:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204742.148725:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062187994624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204742.148733:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204742.148736:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204742.148737:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204742.148739:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6197 00000100:00080000:2.0:1788204742.148748:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204742.148751:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204742.148757:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204742.148767:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062187994752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204742.148774:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204742.148777:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204742.148778:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204742.148781:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6197 00000100:00080000:2.0:1788204742.148784:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204742.148786:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204742.148788:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204742.148798:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187994880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204742.148800:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204742.148864:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204742/real 1788204742] req@ffff8e82894e6300 x1875062187994368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204742.148885:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062187994368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204742.148899:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204742.148903:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204742.148908:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187994368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204742.148939:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187994496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204742.148962:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204742.148977:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204742/real 1788204742] req@ffff8e82894e4a80 x1875062187994496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204742.148991:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062187994496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204742.149008:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204742.149012:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204742.149015:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187994496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204742.149021:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187994624:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204742.149044:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204742.149057:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204742/real 1788204742] req@ffff8e82894e6680 x1875062187994624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204742.149070:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062187994624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204742.149078:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204742.149080:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204742.149083:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187994624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204742.149087:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187994752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204742.149105:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204742.149112:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204742/real 1788204742] req@ffff8e82894e7800 x1875062187994752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204742.149120:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062187994752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204742.149125:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204742.149127:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204742.149130:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187994752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204742.149134:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187994880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204742.149145:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204742.149150:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204742/real 1788204742] req@ffff8e82894e6d80 x1875062187994880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204742.149157:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187994880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204742.149163:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204742.149164:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204742.149166:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187994880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204742.852524:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204743.876514:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204744.900530:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204745.924571:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204746.948551:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204747.268470:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204747.268481:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204747.268486:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204747.268513:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6202 00000100:00080000:1.0:1788204747.268525:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204747.268532:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204747.268581:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062187995008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204747.268606:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204747.268609:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204747.268611:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204747.268614:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6202 00000100:00080000:1.0:1788204747.268616:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204747.268619:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204747.268628:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204747.268638:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062187995136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204747.268649:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204747.268649:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187995008:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204747.268655:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204747.268659:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204747.268663:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6202 00000100:00080000:1.0:1788204747.268666:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204747.268668:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204747.268671:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204747.268678:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062187995264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204747.268685:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204747.268688:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204747.268689:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204747.268691:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6202 00000100:00080000:1.0:1788204747.268693:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204747.268695:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204747.268699:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204747.268706:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062187995392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204747.268711:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204747.268713:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204747.268714:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204747.268716:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6202 00000100:00080000:1.0:1788204747.268719:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204747.268722:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204747.268726:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204747.268734:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062187995520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204747.268743:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204747.268791:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204747/real 1788204747] req@ffff8e828d720e00 x1875062187995008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204747.268813:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062187995008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204747.268826:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204747.268831:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204747.268835:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187995008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204747.268865:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187995136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204747.268886:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204747.268896:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204747/real 1788204747] req@ffff8e828d722680 x1875062187995136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204747.268905:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062187995136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204747.268912:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204747.268914:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204747.268917:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187995136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204747.268920:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187995264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204747.268932:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204747.268939:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204747/real 1788204747] req@ffff8e828d720000 x1875062187995264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204747.268946:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062187995264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204747.268952:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204747.268953:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204747.268955:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187995264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204747.268959:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187995392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204747.268971:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204747.268977:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204747/real 1788204747] req@ffff8e828d721500 x1875062187995392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204747.268984:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062187995392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204747.268990:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204747.268993:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204747.268995:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187995392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204747.268998:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187995520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204747.269008:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204747.269014:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204747/real 1788204747] req@ffff8e828d723800 x1875062187995520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204747.269021:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062187995520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204747.269026:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204747.269028:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204747.269030:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187995520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204747.972502:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204748.996505:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204750.020493:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204751.044502:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204752.068494:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204752.388457:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204752.388468:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204752.388473:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204752.388484:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6207 00000100:00080000:1.0:1788204752.388495:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204752.388503:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204752.388544:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187995648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204752.388570:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204752.388573:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204752.388574:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204752.388578:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6207 00000100:00080000:1.0:1788204752.388581:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204752.388584:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204752.388593:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204752.388608:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187995776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204752.388620:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204752.388627:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788204752.388628:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187995648:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204752.388630:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204752.388640:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6207 00000100:00080000:1.0:1788204752.388644:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204752.388647:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204752.388652:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204752.388661:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187995904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204752.388677:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204752.388679:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204752.388680:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204752.388683:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6207 00000100:00080000:1.0:1788204752.388688:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204752.388690:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204752.388693:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788204752.388698:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204752.388703:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187996032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204752.388710:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204752.388713:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204752.388715:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204752.388718:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6207 00000100:00080000:1.0:1788204752.388720:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204752.388723:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204752.388726:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204752.388733:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187996160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204752.388756:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204752/real 1788204752] req@ffff8e828d722300 x1875062187995648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204752.388775:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187995648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204752.388786:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204752.388791:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204752.388795:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187995648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204752.388820:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187995776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204752.388843:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204752.388857:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204752/real 1788204752] req@ffff8e828d720380 x1875062187995776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204752.388871:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187995776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204752.388881:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204752.388884:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204752.388888:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187995776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204752.388894:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187995904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204752.388910:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204752.388922:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204752/real 1788204752] req@ffff8e828d722d80 x1875062187995904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204752.388935:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187995904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204752.388943:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204752.388946:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204752.388949:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187995904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204752.388955:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187996032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204752.388977:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204752.388989:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204752/real 1788204752] req@ffff8e828d721180 x1875062187996032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204752.389014:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187996032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204752.389023:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204752.389026:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204752.389030:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187996032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204752.389036:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187996160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204752.389056:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204752.389074:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204752/real 1788204752] req@ffff8e828d720a80 x1875062187996160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204752.389085:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187996160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204752.389093:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204752.389096:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204752.389099:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187996160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204753.092484:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204754.116492:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204755.140534:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204756.164457:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204757.188490:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204757.508413:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204757.508421:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204757.508424:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204757.508432:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6212 00000100:00080000:1.0:1788204757.508440:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204757.508445:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204757.508473:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062187996288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204757.508491:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204757.508494:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204757.508495:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204757.508497:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6212 00000100:00080000:1.0:1788204757.508499:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204757.508502:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204757.508508:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204757.508519:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062187996416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204757.508519:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187996288:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204757.508535:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204757.508538:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204757.508539:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204757.508542:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6212 00000100:00080000:1.0:1788204757.508544:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204757.508546:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204757.508549:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204757.508557:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062187996544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204757.508563:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204757.508565:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204757.508566:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204757.508568:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6212 00000100:00080000:1.0:1788204757.508571:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204757.508573:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204757.508576:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788204757.508580:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204757.508585:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062187996672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204757.508593:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204757.508595:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204757.508596:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204757.508599:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6212 00000100:00080000:1.0:1788204757.508601:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204757.508603:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204757.508607:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204757.508616:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062187996800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204757.508620:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204757/real 1788204757] req@ffff8e828d720a80 x1875062187996288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204757.508642:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062187996288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204757.508654:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204757.508657:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204757.508659:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187996288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204757.508678:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187996416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204757.508748:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204757.508758:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204757/real 1788204757] req@ffff8e828d721180 x1875062187996416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204757.508768:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062187996416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204757.508773:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204757.508776:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204757.508778:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187996416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204757.508856:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187996544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204757.508873:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204757.508883:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204757/real 1788204757] req@ffff8e828d722d80 x1875062187996544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204757.508923:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062187996544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204757.508928:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204757.508931:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204757.508933:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187996544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204757.508938:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187996672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204757.508957:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204757.508969:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204757/real 1788204757] req@ffff8e828d720380 x1875062187996672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204757.508980:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062187996672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204757.508987:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204757.508991:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204757.508994:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187996672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204757.508999:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187996800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204757.509013:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204757.509025:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204757/real 1788204757] req@ffff8e828d722300 x1875062187996800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204757.509031:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062187996800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204757.509036:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204757.509037:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204757.509039:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187996800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204758.212472:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204759.236462:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204760.260482:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204761.284436:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204762.308486:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204762.628425:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204762.628436:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204762.628441:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204762.628451:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6217 00000100:00080000:2.0:1788204762.628461:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204762.628469:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204762.628509:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062187996928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204762.628549:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204762.628556:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204762.628558:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204762.628563:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6217 00000100:00080000:2.0:1788204762.628569:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204762.628574:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204762.628583:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204762.628585:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187996928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204762.628597:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062187997056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204762.628607:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204762.628609:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204762.628611:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204762.628614:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6217 00000100:00080000:2.0:1788204762.628617:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204762.628619:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204762.628622:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204762.628631:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062187997184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204762.628638:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204762.628643:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1788204762.628644:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204762.628649:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204762.628656:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6217 00000100:00080000:2.0:1788204762.628661:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204762.628666:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204762.628674:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204762.628684:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204762/real 1788204762] req@ffff8e82894e5f80 x1875062187996928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204762.628691:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062187997312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204762.628706:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204762.628709:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062187996928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204762.628710:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204762.628714:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204762.628722:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6217 00000100:00080000:2.0:1788204762.628728:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204762.628729:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204762.628738:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:3.0:1788204762.628738:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204762.628745:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187996928:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788204762.628746:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204762.628763:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062187997440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204762.628771:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187997056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204762.628795:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204762.628807:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204762/real 1788204762] req@ffff8e82894e4380 x1875062187997056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204762.628828:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062187997056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204762.628838:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204762.628841:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204762.628845:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187997056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204762.628851:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187997184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204762.628874:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204762.628887:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204762/real 1788204762] req@ffff8e82894e5c00 x1875062187997184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204762.628901:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062187997184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204762.628913:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204762.628915:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204762.628918:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187997184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204762.628922:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187997312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204762.628938:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204762.628945:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204762/real 1788204762] req@ffff8e82894e7100 x1875062187997312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204762.628953:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062187997312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204762.628958:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204762.628960:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204762.628962:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187997312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204762.628980:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187997440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204762.628993:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204762.628999:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204762/real 1788204762] req@ffff8e82894e6d80 x1875062187997440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204762.629007:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062187997440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204762.629012:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204762.629013:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204762.629016:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187997440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204763.332494:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204764.356529:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204765.380553:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204766.404484:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204767.428481:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204767.748453:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204767.748462:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204767.748466:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204767.748474:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6222 00000100:00080000:0.0:1788204767.748483:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204767.748489:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204767.748524:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062187997568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204767.748552:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204767.748556:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204767.748559:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204767.748564:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6222 00000100:00080000:0.0:1788204767.748568:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204767.748573:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204767.748583:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204767.748595:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062187997696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204767.748605:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204767.748608:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204767.748609:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204767.748614:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6222 00000100:00100000:3.0:1788204767.748616:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187997568:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204767.748617:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204767.748623:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204767.748629:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204767.748639:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062187997824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204767.748646:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204767.748648:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204767.748650:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204767.748653:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6222 00000100:00080000:0.0:1788204767.748655:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204767.748657:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204767.748661:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204767.748667:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062187997952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204767.748673:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204767.748676:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204767.748677:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204767.748679:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6222 00000100:00080000:0.0:1788204767.748682:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204767.748685:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204767.748687:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204767.748694:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062187998080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204767.748733:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187997696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204767.748765:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187997824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204767.748782:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187997952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204767.748797:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187998080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204768.452493:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204769.476543:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204770.500528:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788204770.820662:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:1.0:1788204770.820721:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788204770.820738:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788204770.820785:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788204770.820799:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788204770.820813:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788204770.820826:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:3.0:1788204770.820831:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204767/real 1788204770] req@ffff8e828793d500 x1875062187997696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204770.820864:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062187997696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204770.820881:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204770.820888:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204770.820894:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187997696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204770.820910:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204767/real 1788204770] req@ffff8e828793fb80 x1875062187997824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204770.820927:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062187997824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204770.820936:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204770.820940:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204770.820944:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187997824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204770.820952:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204767/real 1788204770] req@ffff8e828793d880 x1875062187997952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204770.820965:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062187997952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204770.820972:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204770.820976:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204770.820979:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187997952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204770.820986:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204767/real 1788204770] req@ffff8e828793e300 x1875062187998080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204770.820996:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062187998080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204770.821004:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204770.821007:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204770.821011:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187998080:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204770.821018:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204767/real 1788204770] req@ffff8e828793c380 x1875062187997568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204770.821030:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062187997568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204770.821038:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204770.821042:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204770.821045:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187997568:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788204771.524507:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204772.548543:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204772.868410:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204772.868422:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204772.868426:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204772.868438:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6227 00000100:00080000:0.0:1788204772.868449:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204772.868457:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204772.868492:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187998208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204772.868525:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204772.868529:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204772.868531:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204772.868536:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6227 00000100:00080000:0.0:1788204772.868540:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204772.868550:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204772.868560:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204772.868573:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187998336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204772.868583:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204772.868586:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204772.868588:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204772.868591:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6227 00000100:00080000:0.0:1788204772.868594:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204772.868598:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204772.868603:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204772.868612:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187998464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204772.868621:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204772.868624:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204772.868627:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204772.868630:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6227 00000100:00080000:0.0:1788204772.868634:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204772.868637:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204772.868642:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204772.868651:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187998592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204772.868660:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204772.868663:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204772.868664:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204772.868668:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6227 00000100:00080000:0.0:1788204772.868671:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204772.868675:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204772.868679:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204772.868690:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187998720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204772.868715:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187998208:192.168.201.121@tcp:250:kworker.0 00000800:00000100:3.0:1788204772.868800:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204772.868858:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204772/real 1788204772] req@ffff8e828793f480 x1875062187998208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204772.868882:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187998208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204772.868895:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204772.868899:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204772.868904:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187998208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204772.868912:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187998336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204772.868938:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204772.868953:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204772/real 1788204772] req@ffff8e828793dc00 x1875062187998336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204772.868965:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187998336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204772.868974:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204772.868978:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204772.868981:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187998336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204772.868987:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187998464:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204772.869010:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204772.869022:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204772/real 1788204772] req@ffff8e828793c000 x1875062187998464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204772.869036:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187998464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204772.869045:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204772.869062:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204772.869066:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187998464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204772.869074:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187998592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204772.869098:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204772.869110:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204772/real 1788204772] req@ffff8e828793ca80 x1875062187998592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204772.869122:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187998592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204772.869131:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204772.869135:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204772.869138:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187998592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204772.869144:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187998720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204772.869165:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204772.869177:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204772/real 1788204772] req@ffff8e828793f100 x1875062187998720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204772.869188:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187998720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204772.869201:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204772.869205:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204772.869208:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187998720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204773.572484:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204774.596475:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204775.620441:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204776.644451:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204777.668424:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204777.988377:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204777.988387:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204777.988391:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204777.988400:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6233 00000100:00080000:0.0:1788204777.988410:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204777.988416:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204777.988450:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062187998848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204777.988490:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204777.988494:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204777.988495:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204777.988500:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6233 00000100:00080000:0.0:1788204777.988503:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204777.988508:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204777.988517:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204777.988531:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062187998976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204777.988534:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187998848:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204777.988547:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204777.988549:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204777.988550:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204777.988554:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6233 00000100:00080000:0.0:1788204777.988557:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204777.988561:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204777.988566:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204777.988585:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062187999104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204777.988591:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204777.988602:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204777.988606:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204777.988608:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204777.988612:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6233 00000100:00080000:0.0:1788204777.988617:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204777.988621:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204777.988626:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204777.988633:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204777/real 1788204777] req@ffff8e828793ce00 x1875062187998848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204777.988640:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062187999232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204777.988654:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204777.988657:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062187998848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204777.988658:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204777.988662:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204777.988669:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6233 00000100:00080000:3.0:1788204777.988673:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204777.988675:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204777.988679:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:0.0:1788204777.988682:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204777.988684:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187998848:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788204777.988688:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204777.988700:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062187999360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204777.988712:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187998976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204777.988737:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204777.988749:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204777/real 1788204777] req@ffff8e828793f800 x1875062187998976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204777.988757:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062187998976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204777.988763:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204777.988765:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204777.988767:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187998976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204777.988771:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187999104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204777.988784:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204777.988798:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204777/real 1788204777] req@ffff8e828793e680 x1875062187999104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204777.988805:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062187999104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204777.988810:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204777.988812:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204777.988814:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187999104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204777.988817:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187999232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204777.988827:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204777.988833:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204777/real 1788204777] req@ffff8e828793c700 x1875062187999232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204777.988839:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062187999232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204777.988844:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204777.988845:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204777.988847:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187999232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204777.988850:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187999360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204777.988859:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204777.988863:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204777/real 1788204777] req@ffff8e828793f100 x1875062187999360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204777.988870:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062187999360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204777.988874:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204777.988876:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204777.988878:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187999360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204778.692434:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204779.716475:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204780.740469:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204781.764481:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204782.788463:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204783.108421:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204783.108428:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204783.108431:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204783.108439:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6238 00000100:00080000:0.0:1788204783.108447:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204783.108453:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204783.108488:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062187999488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204783.108507:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204783.108510:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204783.108512:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204783.108516:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6238 00000100:00080000:0.0:1788204783.108520:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204783.108524:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204783.108533:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204783.108545:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062187999616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204783.108555:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187999488:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204783.108571:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204783.108574:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204783.108576:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204783.108580:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6238 00000100:00080000:0.0:1788204783.108583:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204783.108587:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204783.108592:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204783.108604:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062187999744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204783.108621:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788204783.108625:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204783.108627:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204783.108631:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204783.108635:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6238 00000100:00080000:0.0:1788204783.108640:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204783.108645:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204783.108650:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204783.108660:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204783/real 1788204783] req@ffff8e828793ca80 x1875062187999488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204783.108664:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062187999872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204783.108682:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204783.108686:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062187999488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204783.108688:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204783.108691:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204783.108701:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6238 00000100:00080000:0.0:1788204783.108707:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204783.108707:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204783.108716:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:0.0:1788204783.108719:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204783.108723:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062187999488:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788204783.108726:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204783.108739:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062188000000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204783.108749:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187999616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204783.108774:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204783.108786:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204783/real 1788204783] req@ffff8e828793c000 x1875062187999616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204783.108797:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062187999616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204783.108817:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204783.108820:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204783.108823:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187999616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204783.108829:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187999744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204783.108848:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204783.108864:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204783/real 1788204783] req@ffff8e828793dc00 x1875062187999744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204783.108961:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062187999744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204783.108967:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204783.108969:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204783.108971:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187999744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204783.108975:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187999872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204783.108990:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204783.108997:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204783/real 1788204783] req@ffff8e828793f480 x1875062187999872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204783.109004:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062187999872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204783.109008:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204783.109009:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204783.109012:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062187999872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204783.109023:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188000000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204783.109041:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204783.109051:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204783/real 1788204783] req@ffff8e828793c380 x1875062188000000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204783.109062:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062188000000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204783.109069:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204783.109071:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204783.109074:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188000000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204783.813843:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204784.836536:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204785.860532:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204786.884505:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204787.908518:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204788.228421:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204788.228431:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204788.228435:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204788.228444:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6243 00000100:00080000:0.0:1788204788.228454:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204788.228460:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204788.228496:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062188000128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204788.228525:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204788.228529:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204788.228531:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204788.228534:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6243 00000100:00080000:0.0:1788204788.228538:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204788.228541:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204788.228551:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204788.228572:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062188000256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204788.228582:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188000128:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204788.228588:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204788.228593:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204788.228595:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204788.228600:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6243 00000100:00080000:0.0:1788204788.228604:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204788.228609:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204788.228615:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204788.228628:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062188000384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204788.228637:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204788.228640:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204788.228642:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204788.228646:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6243 00000100:00080000:0.0:1788204788.228652:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204788.228656:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:3.0:1788204788.228662:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788204788.228663:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204788.228681:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062188000512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204788.228693:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204788.228696:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204788.228699:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204788.228703:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6243 00000100:00080000:0.0:1788204788.228708:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204788.228712:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204788/real 1788204788] req@ffff8e828793e300 x1875062188000128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204788.228714:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204788.228727:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204788.228745:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062188000128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204788.228747:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062188000640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204788.228804:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204788.228809:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204788.228815:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188000128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204788.228843:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188000256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204788.228863:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204788.228873:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204788/real 1788204788] req@ffff8e828793d880 x1875062188000256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204788.228881:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062188000256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204788.228886:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204788.228888:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204788.228890:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188000256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204788.228894:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188000384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204788.228904:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204788.228913:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204788/real 1788204788] req@ffff8e828793fb80 x1875062188000384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204788.228923:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062188000384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204788.228931:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204788.228934:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204788.228937:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188000384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204788.228944:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188000512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204788.228962:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204788.228970:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204788/real 1788204788] req@ffff8e828793d500 x1875062188000512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204788.228977:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062188000512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204788.228982:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204788.228984:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204788.228986:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188000512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204788.228990:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188000640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204788.229001:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204788.229006:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204788/real 1788204788] req@ffff8e828793c380 x1875062188000640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204788.229013:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062188000640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204788.229017:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204788.229019:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204788.229020:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188000640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204788.932512:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204789.956476:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204790.980444:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204792.004444:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204793.028465:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204793.348423:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204793.348430:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204793.348433:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204793.348442:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6248 00000100:00080000:2.0:1788204793.348454:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204793.348461:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204793.348489:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062188000768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204793.348512:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204793.348515:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204793.348516:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204793.348519:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6248 00000100:00080000:2.0:1788204793.348527:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204793.348531:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204793.348538:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204793.348551:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188000768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204793.348552:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062188000896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204793.348567:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204793.348570:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204793.348571:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204793.348575:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6248 00000100:00080000:2.0:1788204793.348578:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204793.348580:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204793.348583:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204793.348591:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062188001024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204793.348598:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204793.348600:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204793.348601:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204793.348603:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6248 00000100:00080000:2.0:1788204793.348605:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204793.348607:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204793.348610:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204793.348617:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062188001152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788204793.348624:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204793.348632:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204793.348634:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204793.348636:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204793.348639:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6248 00000100:00080000:2.0:1788204793.348641:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204793.348643:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204793.348646:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204793.348652:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062188001280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204793.348666:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204793/real 1788204793] req@ffff8e82894e7800 x1875062188000768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204793.348682:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062188000768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204793.348689:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204793.348693:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204793.348715:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188000768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204793.348735:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188000896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204793.348758:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204793.348777:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204793/real 1788204793] req@ffff8e82894e6680 x1875062188000896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204793.348790:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062188000896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204793.348799:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204793.348801:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204793.348804:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188000896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204793.348808:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188001024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204793.348826:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204793.348835:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204793/real 1788204793] req@ffff8e82894e4a80 x1875062188001024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204793.348842:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062188001024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204793.348847:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204793.348849:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204793.348851:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188001024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204793.348855:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188001152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204793.348867:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204793.348874:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204793/real 1788204793] req@ffff8e82894e6300 x1875062188001152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204793.348881:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062188001152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204793.348886:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204793.348888:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204793.348890:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188001152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204793.348893:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188001280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204793.348904:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204793.348910:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204793/real 1788204793] req@ffff8e82894e4e00 x1875062188001280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204793.348920:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062188001280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204793.348927:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204793.348929:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204793.348934:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188001280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204794.052456:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204795.076448:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204796.100451:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204797.124462:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204798.148491:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204798.468433:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204798.468443:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204798.468448:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204798.468458:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6253 00000100:00080000:0.0:1788204798.468469:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204798.468477:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204798.468512:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062188001408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204798.468538:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204798.468541:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204798.468550:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204798.468554:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6253 00000100:00080000:0.0:1788204798.468558:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204798.468562:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204798.468569:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188001408:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788204798.468571:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204798.468583:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062188001536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204798.468592:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204798.468594:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204798.468596:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204798.468599:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6253 00000100:00080000:0.0:1788204798.468601:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204798.468604:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204798.468629:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204798.468640:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062188001664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204798.468648:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204798.468650:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204798.468652:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788204798.468654:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204798.468655:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6253 00000100:00080000:0.0:1788204798.468658:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204798.468661:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204798.468665:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204798.468672:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062188001792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204798.468680:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204798.468682:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204798.468684:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204798.468687:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6253 00000100:00080000:0.0:1788204798.468690:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204798.468690:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204798/real 1788204798] req@ffff8e828793f480 x1875062188001408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204798.468700:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204798.468705:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204798.468706:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062188001408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204798.468713:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062188001920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204798.468715:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204798.468718:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204798.468721:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188001408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204798.468742:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188001536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204798.468762:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204798.468771:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204798/real 1788204798] req@ffff8e828793dc00 x1875062188001536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204798.468778:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062188001536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204798.468783:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204798.468785:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204798.468787:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188001536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204798.468790:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188001664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204798.468800:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204798.468806:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204798/real 1788204798] req@ffff8e828793c000 x1875062188001664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204798.468812:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062188001664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204798.468816:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204798.468817:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204798.468819:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188001664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204798.468822:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188001792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204798.468831:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204798.468835:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204798/real 1788204798] req@ffff8e828793ca80 x1875062188001792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204798.468842:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062188001792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204798.468846:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204798.468847:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204798.468849:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188001792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204798.468852:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188001920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204798.468860:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204798.468864:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204798/real 1788204798] req@ffff8e828793f100 x1875062188001920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204798.468870:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062188001920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204798.468874:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204798.468876:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204798.468877:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188001920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204799.172506:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204800.196437:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204801.220470:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204802.244467:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204803.268461:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204803.588437:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204803.588448:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204803.588452:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204803.588461:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6258 00000100:00080000:1.0:1788204803.588471:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204803.588477:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204803.588546:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062188002048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204803.588575:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204803.588578:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204803.588580:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204803.588585:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6258 00000100:00080000:1.0:1788204803.588588:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204803.588592:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204803.588602:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204803.588616:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062188002176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204803.588627:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204803.588629:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188002048:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204803.588632:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204803.588636:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204803.588641:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6258 00000100:00080000:1.0:1788204803.588644:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204803.588648:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204803.588652:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204803.588664:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062188002304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204803.588673:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204803.588676:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204803.588678:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204803.588682:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6258 00000100:00080000:1.0:1788204803.588686:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204803.588690:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204803.588696:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788204803.588703:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204803.588708:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062188002432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204803.588719:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204803.588722:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204803.588724:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204803.588727:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6258 00000100:00080000:1.0:1788204803.588731:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204803.588735:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204803.588740:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204803.588743:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204803/real 1788204803] req@ffff8e828d723800 x1875062188002048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204803.588753:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062188002560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204803.588763:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062188002048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204803.588781:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204803.588785:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204803.588789:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188002048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204803.588809:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188002176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204803.588829:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204803.588840:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204803/real 1788204803] req@ffff8e828d721500 x1875062188002176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204803.588850:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062188002176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204803.588870:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204803.588873:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204803.588876:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188002176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204803.588882:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188002304:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204803.588899:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204803.588908:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204803/real 1788204803] req@ffff8e828d720000 x1875062188002304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204803.588918:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062188002304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204803.588925:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204803.588928:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204803.588931:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188002304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204803.588937:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188002432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204803.588952:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204803.588962:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204803/real 1788204803] req@ffff8e828d722680 x1875062188002432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204803.588973:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062188002432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204803.588980:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204803.588983:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204803.588986:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188002432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204803.588991:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188002560:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204803.589005:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204803.589013:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204803/real 1788204803] req@ffff8e828d720e00 x1875062188002560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204803.589023:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062188002560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204803.589028:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204803.589030:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204803.589031:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188002560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204804.292469:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204805.316487:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204806.340491:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204807.364538:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204808.388477:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204808.708486:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204808.708495:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204808.708500:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204808.708510:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6263 00000100:00080000:2.0:1788204808.708522:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204808.708528:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204808.708568:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062188002688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204808.708594:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204808.708597:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204808.708600:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204808.708604:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6263 00000100:00080000:2.0:1788204808.708607:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204808.708611:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204808.708622:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204808.708634:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062188002816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204808.708643:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188002688:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204808.708644:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204808.708647:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204808.708648:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204808.708651:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6263 00000100:00080000:2.0:1788204808.708654:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204808.708657:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204808.708662:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204808.708671:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062188002944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204808.708678:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204808.708681:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204808.708683:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204808.708686:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6263 00000100:00080000:2.0:1788204808.708688:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204808.708691:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204808.708696:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204808.708704:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062188003072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204808.708711:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204808.708714:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204808.708716:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204808.708719:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6263 00000100:00080000:2.0:1788204808.708721:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788204808.708724:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204808.708725:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204808.708728:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204808.708738:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062188003200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204808.708779:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204808/real 1788204808] req@ffff8e82894e5880 x1875062188002688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204808.708799:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062188002688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204808.708807:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204808.708811:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204808.708813:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188002688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204808.708834:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188002816:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204808.708855:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204808.708864:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204808/real 1788204808] req@ffff8e82894e6a00 x1875062188002816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204808.708871:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062188002816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204808.708877:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204808.708878:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204808.708880:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188002816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204808.708883:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188002944:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204808.708893:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204808.708899:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204808/real 1788204808] req@ffff8e82894e7480 x1875062188002944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204808.708905:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062188002944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204808.708913:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204808.708915:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204808.708917:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188002944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204808.708920:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188003072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204808.708930:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204808.708935:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204808/real 1788204808] req@ffff8e82894e4000 x1875062188003072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204808.708941:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062188003072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204808.708945:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204808.708947:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204808.708949:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188003072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204808.708951:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188003200:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204808.708960:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204808.708964:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204808/real 1788204808] req@ffff8e82894e4e00 x1875062188003200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204808.708970:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062188003200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204808.708974:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204808.708976:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204808.708978:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188003200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204809.412511:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204810.436541:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204811.460500:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204812.484466:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204813.508467:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204813.828420:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204813.828448:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204813.828453:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204813.828462:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6268 00000100:00080000:2.0:1788204813.828472:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204813.828480:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204813.828514:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062188003328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204813.828538:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204813.828542:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204813.828544:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204813.828548:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6268 00000100:00080000:2.0:1788204813.828553:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204813.828557:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204813.828566:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204813.828577:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188003328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204813.828581:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062188003456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204813.828598:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204813.828602:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204813.828604:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204813.828608:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6268 00000100:00080000:2.0:1788204813.828612:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204813.828615:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204813.828620:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204813.828631:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062188003584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204813.828639:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204813.828644:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1788204813.828645:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204813.828647:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204813.828654:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6268 00000100:00080000:2.0:1788204813.828657:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204813.828662:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204813.828666:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204813.828678:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062188003712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204813.828686:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204813.828689:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204813.828691:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204813.828695:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6268 00000100:00100000:3.0:1788204813.828700:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204813/real 1788204813] req@ffff8e82894e6300 x1875062188003328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204813.828701:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204813.828709:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204813.828718:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204813.828726:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062188003328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204813.828734:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062188003840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204813.828736:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204813.828743:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204813.828749:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188003328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204813.828771:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188003456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204813.828788:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204813.828796:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204813/real 1788204813] req@ffff8e82894e4a80 x1875062188003456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204813.828803:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062188003456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204813.828809:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204813.828811:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204813.828813:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188003456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204813.828816:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188003584:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204813.828825:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204813.828831:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204813/real 1788204813] req@ffff8e82894e6680 x1875062188003584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204813.828838:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062188003584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204813.828842:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204813.828844:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204813.828845:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188003584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204813.828848:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188003712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204813.828858:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204813.828863:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204813/real 1788204813] req@ffff8e82894e7800 x1875062188003712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204813.828869:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062188003712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204813.828873:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204813.828875:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204813.828876:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188003712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204813.828879:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188003840:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204813.828887:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204813.828891:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204813/real 1788204813] req@ffff8e82894e6d80 x1875062188003840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204813.828896:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062188003840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204813.828901:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204813.828903:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204813.828905:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188003840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204814.532487:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204815.556475:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204816.580474:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204817.604469:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204818.628474:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204818.948436:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204818.948446:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204818.948450:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204818.948460:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6274 00000100:00080000:0.0:1788204818.948470:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204818.948478:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204818.948513:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062188003968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204818.948538:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204818.948541:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204818.948543:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204818.948547:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6274 00000100:00080000:0.0:1788204818.948551:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204818.948556:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204818.948564:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204818.948581:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188003968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204818.948588:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062188004096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204818.948600:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204818.948603:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204818.948605:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204818.948609:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6274 00000100:00080000:0.0:1788204818.948613:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204818.948617:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204818.948622:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204818.948634:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062188004224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204818.948643:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204818.948648:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204818.948653:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204818.948655:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204818.948660:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6274 00000100:00080000:0.0:1788204818.948664:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204818.948668:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204818.948680:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204818.948692:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062188004352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204818.948695:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204818/real 1788204818] req@ffff8e828793c700 x1875062188003968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204818.948725:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204818.948728:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062188003968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204818.948729:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204818.948734:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204818.948743:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6274 00000100:00080000:3.0:1788204818.948747:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204818.948749:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788204818.948754:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:0.0:1788204818.948756:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204818.948761:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188003968:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788204818.948764:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204818.948780:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062188004480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204818.948791:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188004096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204818.948812:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204818.948823:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204818/real 1788204818] req@ffff8e828793e680 x1875062188004096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204818.948834:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062188004096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204818.948842:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204818.948845:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204818.948848:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188004096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204818.948853:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188004224:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204818.948866:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204818.948873:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204818/real 1788204818] req@ffff8e828793f800 x1875062188004224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204818.948883:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062188004224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204818.948890:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204818.948893:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204818.948895:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188004224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204818.948900:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188004352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204818.948913:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204818.948920:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204818/real 1788204818] req@ffff8e828793ce00 x1875062188004352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204818.948930:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062188004352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204818.948937:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204818.948940:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204818.948942:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188004352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204818.948953:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188004480:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204818.948966:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204818.948973:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204818/real 1788204818] req@ffff8e828793f100 x1875062188004480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204818.948983:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062188004480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204818.948990:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204818.948992:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204818.948995:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188004480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204819.652471:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204820.676529:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204821.700531:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204822.724498:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204823.748508:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204824.068425:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204824.068433:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204824.068436:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204824.068444:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6279 00000100:00080000:0.0:1788204824.068453:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204824.068458:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204824.068494:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062188004608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204824.068512:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204824.068515:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204824.068517:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204824.068521:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6279 00000100:00080000:0.0:1788204824.068524:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204824.068528:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204824.068536:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204824.068549:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062188004736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204824.068555:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204824.068557:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204824.068559:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204824.068561:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6279 00000100:00080000:0.0:1788204824.068564:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204824.068568:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204824.068572:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204824.068578:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062188004864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204824.068591:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204824.068593:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204824.068594:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204824.068600:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6279 00000100:00080000:0.0:1788204824.068606:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204824.068608:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188004608:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204824.068610:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204824.068622:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204824.068637:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062188004992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204824.068643:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204824.068645:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204824.068646:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204824.068648:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6279 00000100:00080000:0.0:1788204824.068650:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204824.068652:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204824.068655:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204824.068661:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062188005120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204824.068700:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204824.068750:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204824/real 1788204824] req@ffff8e828793ca80 x1875062188004608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204824.068766:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062188004608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204824.068776:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204824.068780:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204824.068783:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188004608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204824.068791:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188004736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204824.068813:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204824.068826:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204824/real 1788204824] req@ffff8e828793c000 x1875062188004736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204824.068837:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062188004736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204824.068850:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204824.068853:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204824.068856:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188004736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204824.068882:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188004864:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204824.068901:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204824.068912:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204824/real 1788204824] req@ffff8e828793dc00 x1875062188004864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204824.068923:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062188004864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204824.068931:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204824.068934:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204824.068937:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188004864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204824.068943:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188004992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204824.068962:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204824.068973:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204824/real 1788204824] req@ffff8e828793f480 x1875062188004992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204824.068984:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062188004992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204824.068991:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204824.068994:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204824.068998:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188004992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204824.069004:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188005120:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204824.069023:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204824.069033:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204824/real 1788204824] req@ffff8e828793c380 x1875062188005120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204824.069044:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062188005120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204824.069051:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204824.069055:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204824.069058:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188005120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204824.772500:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204825.796512:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204826.820449:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204827.844499:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204828.868490:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204829.188450:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204829.188456:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204829.188460:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204829.188467:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6284 00000100:00080000:0.0:1788204829.188476:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204829.188482:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204829.188509:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062188005248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204829.188540:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204829.188543:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204829.188545:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204829.188548:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6284 00000100:00080000:0.0:1788204829.188551:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204829.188555:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204829.188564:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204829.188574:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188005248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204829.188581:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062188005376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204829.188595:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204829.188599:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204829.188601:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204829.188606:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6284 00000100:00080000:0.0:1788204829.188610:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204829.188615:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204829.188621:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204829.188632:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062188005504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204829.188642:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788204829.188646:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204829.188647:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204829.188651:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204829.188656:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6284 00000100:00080000:0.0:1788204829.188661:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204829.188665:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204829.188670:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204829.188682:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062188005632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204829.188690:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204829.188695:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788204829.188695:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204829/real 1788204829] req@ffff8e828793d500 x1875062188005248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204829.188699:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204829.188707:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6284 00000100:00080000:0.0:1788204829.188712:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204829.188718:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204829.188723:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204829.188724:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062188005248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204829.188737:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062188005760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204829.188738:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204829.188747:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204829.188754:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188005248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204829.188777:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188005376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204829.188796:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204829.188816:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204829/real 1788204829] req@ffff8e828793fb80 x1875062188005376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204829.188828:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062188005376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204829.188836:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204829.188839:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204829.188843:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188005376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204829.188849:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188005504:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204829.188863:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204829.188872:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204829/real 1788204829] req@ffff8e828793d880 x1875062188005504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204829.188878:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062188005504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204829.188883:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204829.188885:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204829.188887:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188005504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204829.188890:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188005632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204829.188901:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204829.188907:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204829/real 1788204829] req@ffff8e828793e300 x1875062188005632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204829.188913:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062188005632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204829.188917:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204829.188919:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204829.188921:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188005632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204829.188923:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188005760:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204829.188933:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204829.188939:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204829/real 1788204829] req@ffff8e828793c380 x1875062188005760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204829.188947:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062188005760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204829.188952:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204829.188954:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204829.188955:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188005760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204829.892534:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204830.916521:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204831.940478:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204832.964520:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204833.988476:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204834.308443:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204834.308455:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204834.308460:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204834.308473:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6289 00000100:00080000:2.0:1788204834.308485:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204834.308493:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204834.308540:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062188005888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204834.308573:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204834.308577:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204834.308579:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204834.308583:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6289 00000100:00080000:2.0:1788204834.308587:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204834.308591:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204834.308603:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204834.308622:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062188006016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204834.308634:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188005888:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204834.308635:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204834.308644:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204834.308647:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204834.308651:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6289 00000100:00080000:2.0:1788204834.308654:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204834.308658:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204834.308664:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204834.308674:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062188006144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204834.308683:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204834.308687:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204834.308689:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204834.308693:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6289 00000100:00080000:2.0:1788204834.308697:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204834.308702:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204834.308708:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204834.308719:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062188006272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204834.308728:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204834.308732:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204834.308734:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204834.308737:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6289 00000100:00080000:2.0:1788204834.308743:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204834.308756:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204834.308763:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204834.308772:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188006016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204834.308782:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062188006400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204834.308814:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188006144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204834.308840:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188006272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204834.308874:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188006400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204835.012516:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204836.036467:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204837.060512:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1788204837.380684:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:0.0:1788204837.380743:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788204837.380763:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:0.0:1788204837.380828:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:0.0:1788204837.380839:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:0.0:1788204837.380848:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:0.0:1788204837.380858:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:3.0:1788204837.380995:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204834/real 1788204837] req@ffff8e82894e6d80 x1875062188006400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204837.381027:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062188006400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204837.381048:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204837.381054:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204837.381059:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188006400:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204837.381068:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204834/real 1788204837] req@ffff8e82894e5c00 x1875062188006016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204837.381081:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062188006016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204837.381090:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204837.381094:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204837.381097:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188006016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204837.381103:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204834/real 1788204837] req@ffff8e82894e4380 x1875062188006144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204837.381115:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062188006144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204837.381122:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204837.381126:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204837.381129:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188006144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204837.381136:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204834/real 1788204837] req@ffff8e82894e5f80 x1875062188006272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204837.381145:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062188006272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204837.381152:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204837.381155:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204837.381158:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188006272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204837.381164:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204834/real 1788204837] req@ffff8e82894e7100 x1875062188005888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204837.381174:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062188005888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204837.381180:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204837.381183:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204837.381185:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188005888:192.168.201.121@tcp:250:kworker.0 00010000:00010000:0.0:1788204838.084486:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204839.108504:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204839.428436:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204839.428446:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204839.428450:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204839.428461:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6294 00000100:00080000:2.0:1788204839.428473:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204839.428480:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204839.428523:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062188006528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204839.428554:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204839.428558:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204839.428559:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204839.428563:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6294 00000100:00080000:2.0:1788204839.428567:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204839.428580:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204839.428595:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204839.428606:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188006528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204839.428613:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062188006656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204839.428625:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204839.428631:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204839.428633:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204839.428637:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6294 00000100:00080000:2.0:1788204839.428641:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204839.428646:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204839.428651:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204839.428663:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062188006784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204839.428671:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204839.428674:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204839.428675:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204839.428679:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6294 00000100:00080000:2.0:1788204839.428683:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204839.428692:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204839.428699:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788204839.428702:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204839.428716:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062188006912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204839.428726:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204839.428730:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204839.428732:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204839.428741:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6294 00000100:00080000:2.0:1788204839.428745:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204839.428751:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204839.428757:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204839.428769:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062188007040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204839.428773:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204839/real 1788204839] req@ffff8e82894e7800 x1875062188006528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204839.428805:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062188006528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204839.428818:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204839.428823:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204839.428828:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188006528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204839.428866:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188006656:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204839.428896:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204839.428945:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204839/real 1788204839] req@ffff8e82894e6680 x1875062188006656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204839.428958:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062188006656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204839.428966:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204839.428970:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204839.428973:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188006656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204839.428979:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188006784:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204839.429017:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204839.429040:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204839/real 1788204839] req@ffff8e82894e4a80 x1875062188006784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204839.429053:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062188006784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204839.429061:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204839.429065:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204839.429069:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188006784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204839.429076:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188006912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204839.429098:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204839.429109:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204839/real 1788204839] req@ffff8e82894e6300 x1875062188006912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204839.429126:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062188006912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204839.429134:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204839.429137:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204839.429140:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188006912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204839.429146:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188007040:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204839.429166:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204839.429176:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204839/real 1788204839] req@ffff8e82894e4e00 x1875062188007040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204839.429187:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062188007040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204839.429195:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204839.429198:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204839.429201:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188007040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204840.132453:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204841.156517:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204842.180490:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204843.204421:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204844.228476:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204844.548425:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204844.548432:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204844.548435:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204844.548443:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6299 00000100:00080000:0.0:1788204844.548453:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204844.548458:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204844.548486:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062188007168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204844.548505:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204844.548508:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204844.548509:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204844.548512:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6299 00000100:00080000:0.0:1788204844.548514:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204844.548515:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204844.548522:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204844.548532:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062188007296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204844.548542:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204844.548545:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204844.548548:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204844.548551:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6299 00000100:00080000:0.0:1788204844.548555:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204844.548559:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204844.548564:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204844.548576:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062188007424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204844.548587:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204844.548593:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788204844.548596:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188007168:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204844.548597:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204844.548607:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6299 00000100:00080000:0.0:1788204844.548613:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204844.548618:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204844.548623:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204844.548637:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062188007552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204844.548646:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204844.548649:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204844.548651:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204844.548655:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6299 00000100:00080000:0.0:1788204844.548661:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204844.548665:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204844.548669:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204844.548681:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062188007680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204844.548693:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204844.548749:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204844/real 1788204844] req@ffff8e828793f480 x1875062188007168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204844.548767:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062188007168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204844.548785:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204844.548789:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204844.548794:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188007168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204844.548802:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188007296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204844.548837:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204844.548849:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204844/real 1788204844] req@ffff8e828793dc00 x1875062188007296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204844.548861:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062188007296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204844.548869:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204844.548872:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204844.548875:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188007296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204844.548904:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188007424:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204844.548922:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204844.548932:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204844/real 1788204844] req@ffff8e828793c000 x1875062188007424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204844.548942:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062188007424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204844.548950:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204844.548953:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204844.548956:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188007424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204844.548972:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188007552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204844.548991:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204844.549001:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204844/real 1788204844] req@ffff8e828793ca80 x1875062188007552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204844.549011:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062188007552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204844.549018:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204844.549021:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204844.549024:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188007552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204844.549030:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188007680:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204844.549046:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204844.549055:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204844/real 1788204844] req@ffff8e828793f100 x1875062188007680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204844.549066:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062188007680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204844.549073:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204844.549076:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204844.549079:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188007680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204845.252517:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204846.276481:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204847.300512:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204848.324509:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204849.348465:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204849.668451:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204849.668461:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204849.668466:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204849.668475:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6304 00000100:00080000:0.0:1788204849.668486:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204849.668493:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204849.668528:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062188007808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204849.668556:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204849.668560:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204849.668562:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204849.668566:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6304 00000100:00080000:0.0:1788204849.668569:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204849.668574:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204849.668582:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204849.668596:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062188007936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204849.668606:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188007808:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204849.668608:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204849.668615:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204849.668617:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204849.668620:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6304 00000100:00080000:0.0:1788204849.668624:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204849.668627:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204849.668632:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204849.668644:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062188008064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204849.668653:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204849.668656:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204849.668658:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788204849.668662:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204849.668667:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6304 00000100:00080000:0.0:1788204849.668672:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204849.668677:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204849.668682:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204849.668696:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062188008192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204849.668705:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204849/real 1788204849] req@ffff8e828793ce00 x1875062188007808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204849.668706:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204849.668717:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204849.668721:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204849.668741:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6304 00000100:00080000:0.0:1788204849.668744:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204849.668750:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204849.668755:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062188007808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788204849.668756:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788204849.668768:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204849.668771:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204849.668776:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188007808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204849.668777:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062188008320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204849.668797:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188007936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204849.668815:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204849.668822:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204849/real 1788204849] req@ffff8e828793f800 x1875062188007936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204849.668829:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062188007936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204849.668835:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204849.668836:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204849.668838:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188007936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204849.668841:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188008064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204849.668850:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204849.668865:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204849/real 1788204849] req@ffff8e828793e680 x1875062188008064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204849.668872:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062188008064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204849.668880:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204849.668882:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204849.668884:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188008064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204849.668887:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188008192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204849.668898:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204849.668903:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204849/real 1788204849] req@ffff8e828793c700 x1875062188008192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204849.668909:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062188008192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204849.668913:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204849.668915:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204849.668916:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188008192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204849.668919:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188008320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204849.668927:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204849.668932:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204849/real 1788204849] req@ffff8e828793f100 x1875062188008320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204849.668938:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062188008320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204849.668942:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204849.668944:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204849.668946:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188008320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204850.372494:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204851.396456:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204852.420480:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204853.444484:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204854.468482:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204854.788494:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204854.788501:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204854.788504:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204854.788512:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6309 00000100:00080000:2.0:1788204854.788523:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204854.788538:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204854.788569:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062188008448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204854.788592:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204854.788594:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204854.788595:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204854.788598:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6309 00000100:00080000:2.0:1788204854.788600:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204854.788602:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204854.788608:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204854.788619:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062188008576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204854.788628:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204854.788631:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204854.788642:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204854.788646:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6309 00000100:00080000:2.0:1788204854.788650:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204854.788655:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204854.788660:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204854.788671:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188008448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204854.788678:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062188008704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204854.788691:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204854.788693:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204854.788695:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204854.788697:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6309 00000100:00080000:2.0:1788204854.788699:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204854.788702:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204854.788705:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204854.788713:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062188008832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204854.788718:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204854.788721:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204854.788722:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204854.788725:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6309 00000100:00080000:2.0:1788204854.788726:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204854.788729:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204854.788731:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204854.788743:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062188008960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204854.788777:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204854.788832:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204854/real 1788204854] req@ffff8e82894e4000 x1875062188008448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204854.788852:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062188008448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204854.788864:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204854.788868:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204854.788873:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188008448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204854.788881:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188008576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204854.788905:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204854.788918:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204854/real 1788204854] req@ffff8e82894e7480 x1875062188008576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204854.788930:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062188008576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204854.788938:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204854.788942:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204854.788945:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188008576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204854.788976:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188008704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204854.788997:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204854.789008:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204854/real 1788204854] req@ffff8e82894e6a00 x1875062188008704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204854.789019:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062188008704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204854.789027:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204854.789030:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204854.789034:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188008704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204854.789040:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188008832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204854.789061:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204854.789071:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204854/real 1788204854] req@ffff8e82894e5880 x1875062188008832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204854.789083:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062188008832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204854.789090:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204854.789094:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204854.789096:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188008832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204854.789102:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188008960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204854.789121:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204854.789131:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204854/real 1788204854] req@ffff8e82894e4e00 x1875062188008960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204854.789138:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062188008960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204854.789143:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204854.789145:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204854.789147:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188008960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204855.492486:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204856.516434:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204857.540492:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204858.564425:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204859.588537:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204859.908428:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204859.908436:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204859.908440:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204859.908449:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6314 00000100:00080000:2.0:1788204859.908457:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204859.908463:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204859.908494:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062188009088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204859.908513:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204859.908515:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204859.908516:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204859.908519:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6314 00000100:00080000:2.0:1788204859.908521:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204859.908524:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204859.908532:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204859.908556:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062188009216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204859.908567:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204859.908570:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204859.908572:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204859.908575:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6314 00000100:00080000:2.0:1788204859.908579:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204859.908583:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204859.908588:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204859.908597:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062188009344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204859.908605:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204859.908607:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204859.908609:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204859.908617:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6314 00000100:00100000:3.0:1788204859.908619:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188009088:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204859.908620:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204859.908629:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204859.908632:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204859.908645:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062188009472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204859.908652:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204859.908654:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204859.908655:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204859.908658:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6314 00000100:00080000:2.0:1788204859.908661:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204859.908663:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204859.908666:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204859.908673:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062188009600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204859.908715:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204859.908767:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204859/real 1788204859] req@ffff8e82894e6300 x1875062188009088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204859.908790:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062188009088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204859.908803:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204859.908807:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204859.908811:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188009088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204859.908820:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188009216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204859.908845:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204859.908858:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204859/real 1788204859] req@ffff8e82894e4a80 x1875062188009216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204859.908870:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062188009216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204859.908879:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204859.908882:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204859.908887:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188009216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204859.908925:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188009344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204859.908945:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204859.908955:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204859/real 1788204859] req@ffff8e82894e6680 x1875062188009344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204859.908965:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062188009344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204859.908970:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204859.908972:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204859.908974:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188009344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204859.908978:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188009472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204859.908992:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204859.908998:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204859/real 1788204859] req@ffff8e82894e7800 x1875062188009472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204859.909004:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062188009472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204859.909009:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204859.909011:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204859.909012:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188009472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204859.909016:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188009600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204859.909052:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204859.909063:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204859/real 1788204859] req@ffff8e82894e7100 x1875062188009600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204859.909076:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062188009600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204859.909084:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204859.909087:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204859.909090:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188009600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204860.612532:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204861.636442:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204862.660489:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204863.684512:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204864.708488:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204865.028436:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204865.028446:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204865.028450:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204865.028473:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6320 00000100:00080000:0.0:1788204865.028483:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204865.028491:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204865.028528:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062188009728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204865.028560:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204865.028564:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204865.028566:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204865.028571:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6320 00000100:00080000:0.0:1788204865.028575:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204865.028580:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204865.028591:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204865.028599:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188009728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204865.028608:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062188009856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204865.028628:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204865.028632:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204865.028634:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204865.028638:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6320 00000100:00080000:0.0:1788204865.028643:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204865.028647:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204865.028654:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204865.028667:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062188009984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204865.028677:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204865.028682:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1788204865.028683:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204865.028685:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204865.028691:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6320 00000100:00080000:0.0:1788204865.028696:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204865.028701:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204865.028707:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204865.028720:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062188010112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204865.028729:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204865.028733:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204865.028735:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788204865.028738:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204865/real 1788204865] req@ffff8e828793ca80 x1875062188009728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204865.028751:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6320 00000100:00080000:0.0:1788204865.028757:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788204865.028759:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062188009728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204865.028764:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:2.0:1788204865.028770:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788204865.028772:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788204865.028775:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204865.028780:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188009728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204865.028790:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062188010240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788204865.028804:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188009856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204865.028823:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204865.028833:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204865/real 1788204865] req@ffff8e828793c000 x1875062188009856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204865.028842:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062188009856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204865.028848:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204865.028850:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204865.028852:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188009856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204865.028855:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188009984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:2.0:1788204865.028866:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204865.028872:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204865/real 1788204865] req@ffff8e828793dc00 x1875062188009984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204865.028880:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062188009984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204865.028889:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204865.028891:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204865.028893:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188009984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:2.0:1788204865.028896:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188010112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204865.028907:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204865.028913:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204865/real 1788204865] req@ffff8e828793f480 x1875062188010112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204865.028920:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062188010112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204865.028925:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204865.028927:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204865.028929:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188010112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:2.0:1788204865.028933:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188010240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:2.0:1788204865.028949:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204865.028961:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204865/real 1788204865] req@ffff8e828793c380 x1875062188010240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204865.028975:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062188010240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204865.028983:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788204865.028987:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204865.028992:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188010240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204865.732513:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204866.756501:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204867.780497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204868.804473:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204869.828504:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204870.148422:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204870.148436:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204870.148441:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204870.148453:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6325 00000100:00080000:2.0:1788204870.148466:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204870.148479:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204870.148529:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062188010368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204870.148566:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204870.148569:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204870.148572:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204870.148576:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6325 00000100:00080000:2.0:1788204870.148580:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204870.148585:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204870.148596:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204870.148602:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188010368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204870.148613:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062188010496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204870.148625:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204870.148629:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204870.148630:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204870.148634:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6325 00000100:00080000:2.0:1788204870.148638:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204870.148643:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204870.148648:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204870.148661:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062188010624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204870.148670:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204870.148674:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204870.148676:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204870.148680:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6325 00000800:00000100:3.0:1788204870.148685:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204870.148686:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204870.148694:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204870.148700:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204870.148711:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062188010752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204870.148721:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204870.148724:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204870.148726:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204870.148729:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6325 00000100:00080000:2.0:1788204870.148732:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204870.148736:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204870.148741:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204870.148744:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204870/real 1788204870] req@ffff8e82894e5f80 x1875062188010368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204870.148757:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062188010880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204870.148768:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062188010368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204870.148781:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204870.148786:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204870.148791:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188010368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204870.148814:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188010496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204870.148832:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204870.148842:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204870/real 1788204870] req@ffff8e82894e4380 x1875062188010496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204870.148853:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062188010496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204870.148875:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204870.148878:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204870.148882:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188010496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204870.148887:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188010624:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204870.148902:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204870.148911:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204870/real 1788204870] req@ffff8e82894e5c00 x1875062188010624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204870.148921:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062188010624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204870.148929:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204870.148932:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204870.148935:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188010624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204870.148940:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188010752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204870.148954:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204870.148969:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204870/real 1788204870] req@ffff8e82894e6d80 x1875062188010752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204870.148980:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062188010752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204870.148987:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204870.148990:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204870.148994:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188010752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204870.148999:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188010880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204870.149013:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204870.149021:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204870/real 1788204870] req@ffff8e82894e7100 x1875062188010880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204870.149031:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062188010880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204870.149039:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204870.149041:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204870.149044:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188010880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204870.852521:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204871.876511:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204872.900474:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204873.924542:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204874.948470:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204875.268422:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204875.268434:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204875.268437:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204875.268445:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6330 00000100:00080000:1.0:1788204875.268453:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204875.268459:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204875.268488:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062188011008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204875.268510:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204875.268513:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204875.268514:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204875.268518:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6330 00000100:00080000:1.0:1788204875.268521:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204875.268525:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204875.268533:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204875.268546:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062188011136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204875.268561:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204875.268564:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204875.268567:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788204875.268568:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188011008:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204875.268573:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6330 00000100:00080000:1.0:1788204875.268579:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204875.268583:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204875.268587:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204875.268599:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062188011264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204875.268607:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204875.268610:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204875.268612:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204875.268615:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6330 00000100:00080000:1.0:1788204875.268618:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204875.268621:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204875.268625:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204875.268635:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062188011392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204875.268639:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:1.0:1788204875.268649:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204875.268652:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204875.268654:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204875.268658:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6330 00000100:00080000:1.0:1788204875.268663:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204875.268666:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204875.268670:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204875.268681:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062188011520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204875.268689:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204875/real 1788204875] req@ffff8e828d720e00 x1875062188011008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204875.268710:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062188011008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204875.268721:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204875.268724:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204875.268728:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188011008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204875.268752:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188011136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204875.268780:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204875.268794:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204875/real 1788204875] req@ffff8e828d722680 x1875062188011136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204875.268806:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062188011136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204875.268814:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204875.268817:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204875.268821:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188011136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204875.268826:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188011264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204875.268843:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204875.268854:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204875/real 1788204875] req@ffff8e828d720000 x1875062188011264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204875.268865:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062188011264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204875.268873:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204875.268876:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204875.268879:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188011264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204875.268885:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188011392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204875.268902:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204875.268912:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204875/real 1788204875] req@ffff8e828d721500 x1875062188011392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204875.268923:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062188011392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204875.268931:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204875.268934:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204875.268937:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188011392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204875.268942:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188011520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204875.268959:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204875.268970:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204875/real 1788204875] req@ffff8e828d723800 x1875062188011520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204875.268981:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062188011520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204875.268988:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204875.268991:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204875.268994:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188011520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204875.972511:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204876.996456:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204878.020443:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204879.044457:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204880.068432:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204880.388416:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204880.388425:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204880.388429:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204880.388438:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6335 00000100:00080000:0.0:1788204880.388447:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204880.388454:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204880.388488:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062188011648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204880.388513:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204880.388517:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204880.388519:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204880.388524:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6335 00000100:00080000:0.0:1788204880.388528:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204880.388533:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204880.388543:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204880.388560:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062188011776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204880.388570:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204880.388574:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788204880.388574:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188011648:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204880.388577:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204880.388584:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6335 00000100:00080000:0.0:1788204880.388587:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204880.388591:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204880.388595:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204880.388605:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062188011904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204880.388612:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204880.388614:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204880.388615:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204880.388618:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6335 00000100:00080000:0.0:1788204880.388620:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204880.388622:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204880.388625:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204880.388632:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062188012032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204880.388638:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204880.388640:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204880.388641:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204880.388644:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6335 00000100:00080000:0.0:1788204880.388646:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204880.388649:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204880.388654:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788204880.388654:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204880.388665:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062188012160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204880.388695:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204880/real 1788204880] req@ffff8e828793e300 x1875062188011648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204880.388709:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062188011648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204880.388716:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204880.388719:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204880.388722:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188011648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204880.388742:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188011776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204880.388758:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204880.388766:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204880/real 1788204880] req@ffff8e828793d880 x1875062188011776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204880.388775:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062188011776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204880.388780:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204880.388782:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204880.388784:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188011776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204880.388788:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188011904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204880.388801:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204880.388811:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204880/real 1788204880] req@ffff8e828793fb80 x1875062188011904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204880.388823:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062188011904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204880.388832:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204880.388836:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204880.388840:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188011904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204880.388846:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188012032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204880.388864:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204880.388872:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204880/real 1788204880] req@ffff8e828793d500 x1875062188012032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204880.388880:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062188012032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204880.388885:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204880.388887:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204880.388889:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188012032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204880.388892:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188012160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204880.388903:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204880.388908:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204880/real 1788204880] req@ffff8e828793c380 x1875062188012160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204880.388915:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062188012160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204880.388920:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204880.388922:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204880.388924:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188012160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204881.092516:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204882.116496:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204883.140487:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204884.164491:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204885.188500:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204885.508421:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204885.508430:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204885.508433:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204885.508442:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6340 00000100:00080000:2.0:1788204885.508450:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204885.508457:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204885.508495:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062188012288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204885.508522:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204885.508526:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204885.508528:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204885.508533:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6340 00000100:00080000:2.0:1788204885.508546:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204885.508551:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204885.508562:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204885.508567:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188012288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204885.508578:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062188012416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204885.508587:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204885.508589:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204885.508591:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204885.508594:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6340 00000100:00080000:2.0:1788204885.508597:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204885.508600:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204885.508604:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204885.508611:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062188012544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204885.508619:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204885.508621:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204885.508625:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788204885.508626:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204885.508629:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6340 00000100:00080000:2.0:1788204885.508633:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204885.508636:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204885.508639:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204885.508647:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062188012672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204885.508653:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204885.508655:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204885.508657:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204885.508660:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6340 00000100:00080000:2.0:1788204885.508662:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204885.508664:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204885.508666:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204885.508675:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204885/real 1788204885] req@ffff8e82894e7800 x1875062188012288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204885.508676:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062188012800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204885.508708:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062188012288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204885.508722:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204885.508725:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204885.508728:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188012288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204885.508747:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188012416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204885.508763:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204885.508772:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204885/real 1788204885] req@ffff8e82894e6680 x1875062188012416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204885.508780:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062188012416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204885.508785:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204885.508787:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204885.508790:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188012416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204885.508793:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188012544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204885.508803:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204885.508810:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204885/real 1788204885] req@ffff8e82894e4a80 x1875062188012544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204885.508817:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062188012544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204885.508822:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204885.508824:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204885.508825:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188012544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204885.508830:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188012672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204885.508840:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204885.508846:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204885/real 1788204885] req@ffff8e82894e6300 x1875062188012672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204885.508853:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062188012672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204885.508858:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204885.508860:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204885.508862:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188012672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204885.508865:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188012800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204885.508875:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204885.508881:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204885/real 1788204885] req@ffff8e82894e4e00 x1875062188012800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204885.508888:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062188012800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204885.508893:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204885.508894:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204885.508896:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188012800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204886.212508:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204887.236520:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204888.260515:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204889.284528:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204890.308481:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204890.628449:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204890.628461:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204890.628465:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204890.628475:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6345 00000100:00080000:0.0:1788204890.628487:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204890.628494:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204890.628540:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062188012928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204890.628575:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204890.628579:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204890.628581:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204890.628586:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6345 00000100:00080000:0.0:1788204890.628589:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204890.628594:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204890.628605:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204890.628627:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062188013056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204890.628638:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204890.628641:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204890.628643:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204890.628651:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6345 00000100:00100000:3.0:1788204890.628653:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188012928:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204890.628657:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204890.628667:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204890.628672:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204890.628683:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062188013184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204890.628692:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204890.628695:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204890.628697:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204890.628701:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6345 00000100:00080000:0.0:1788204890.628704:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204890.628709:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204890.628714:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788204890.628722:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204890.628727:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062188013312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204890.628738:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204890.628741:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204890.628743:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204890.628747:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6345 00000100:00080000:0.0:1788204890.628752:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204890.628756:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204890.628761:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204890.628767:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204890/real 1788204890] req@ffff8e828793f480 x1875062188012928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204890.628776:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062188013440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204890.628825:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062188012928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204890.628844:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204890.628850:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204890.628854:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188012928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204890.628876:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188013056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204890.628902:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204890.628915:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204890/real 1788204890] req@ffff8e828793dc00 x1875062188013056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204890.628935:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062188013056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204890.628945:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204890.628949:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204890.628953:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188013056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204890.628960:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188013184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204890.628982:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204890.628995:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204890/real 1788204890] req@ffff8e828793c000 x1875062188013184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204890.629009:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062188013184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204890.629040:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204890.629045:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204890.629049:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188013184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204890.629056:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188013312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204890.629080:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204890.629091:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204890/real 1788204890] req@ffff8e828793ca80 x1875062188013312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204890.629105:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062188013312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204890.629111:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204890.629113:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204890.629115:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188013312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204890.629120:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188013440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204890.629133:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204890.629140:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204890/real 1788204890] req@ffff8e828793f100 x1875062188013440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204890.629148:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062188013440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204890.629152:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204890.629154:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204890.629157:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188013440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204891.332477:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204892.356480:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204893.380513:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204894.404515:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204895.428536:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204895.748457:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204895.748468:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204895.748473:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204895.748483:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6350 00000100:00080000:2.0:1788204895.748496:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204895.748503:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204895.748548:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062188013568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204895.748577:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204895.748580:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204895.748581:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204895.748585:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6350 00000100:00080000:2.0:1788204895.748589:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204895.748594:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204895.748604:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204895.748612:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062188013696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204895.748621:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788204895.748622:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188013568:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204895.748623:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204895.748627:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204895.748634:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6350 00000100:00080000:2.0:1788204895.748636:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204895.748638:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204895.748640:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204895.748647:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062188013824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204895.748652:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204895.748655:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204895.748657:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204895.748661:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6350 00000100:00080000:2.0:1788204895.748665:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204895.748669:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204895.748675:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204895.748686:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062188013952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204895.748696:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204895.748700:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204895.748701:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788204895.748707:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204895.748709:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6350 00000100:00080000:2.0:1788204895.748714:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204895.748717:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204895.748721:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204895.748732:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062188014080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204895.748762:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204895/real 1788204895] req@ffff8e82894e5880 x1875062188013568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204895.748777:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062188013568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204895.748786:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204895.748790:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204895.748794:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188013568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204895.748816:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188013696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204895.748832:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204895.748840:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204895/real 1788204895] req@ffff8e82894e6a00 x1875062188013696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204895.748847:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062188013696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204895.748852:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204895.748855:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204895.748857:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188013696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204895.748860:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188013824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204895.748869:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204895.748874:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204895/real 1788204895] req@ffff8e82894e7480 x1875062188013824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204895.748885:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062188013824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204895.748889:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204895.748891:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204895.748892:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188013824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204895.748895:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188013952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204895.748906:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204895.748911:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204895/real 1788204895] req@ffff8e82894e4000 x1875062188013952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204895.748921:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062188013952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204895.748939:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204895.748943:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204895.748946:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188013952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204895.748951:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188014080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204895.748971:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204895.748980:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204895/real 1788204895] req@ffff8e82894e4e00 x1875062188014080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204895.748987:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062188014080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204895.748996:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204895.748998:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204895.749000:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188014080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204896.452486:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204897.476473:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204898.500524:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204899.524516:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204900.548566:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204900.868496:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204900.868503:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204900.868507:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204900.868517:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6355 00000100:00080000:0.0:1788204900.868527:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204900.868534:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204900.868565:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062188014208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204900.868595:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204900.868600:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204900.868602:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204900.868609:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6355 00000100:00080000:0.0:1788204900.868615:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204900.868622:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204900.868633:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204900.868642:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188014208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204900.868654:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062188014336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204900.868666:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204900.868670:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204900.868672:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204900.868680:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6355 00000100:00080000:0.0:1788204900.868686:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204900.868691:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204900.868697:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204900.868711:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062188014464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204900.868719:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204900.868722:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204900.868724:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204900.868729:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6355 00000100:00080000:0.0:1788204900.868732:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204900.868736:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204900.868740:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204900.868747:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062188014592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204900.868753:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204900.868756:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204900.868759:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204900.868764:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6355 00000100:00080000:0.0:1788204900.868769:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204900.868772:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204900.868776:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204900.868785:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188014336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204900.868791:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062188014720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204900.868824:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188014464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204900.868873:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188014592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788204900.868917:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188014720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204901.572506:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204902.596572:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204903.620509:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788204903.940624:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788204903.940674:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788204903.940690:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204903.940738:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204903.940750:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204903.940760:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204903.940769:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788204903.940919:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204900/real 1788204903] req@ffff8e828793f100 x1875062188014720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204903.940969:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062188014720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204903.940985:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204903.940992:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204903.940997:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188014720:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788204903.941008:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204900/real 1788204903] req@ffff8e828793e680 x1875062188014336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204903.941023:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062188014336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204903.941032:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204903.941036:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204903.941040:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188014336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204903.941049:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204900/real 1788204903] req@ffff8e828793f800 x1875062188014464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204903.941067:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062188014464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204903.941077:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204903.941081:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204903.941085:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188014464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204903.941093:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204900/real 1788204903] req@ffff8e828793ce00 x1875062188014592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204903.941106:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062188014592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204903.941114:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204903.941118:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204903.941121:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188014592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788204903.941129:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204900/real 1788204903] req@ffff8e828793c700 x1875062188014208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204903.941141:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062188014208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204903.941149:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204903.941153:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204903.941156:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188014208:192.168.201.121@tcp:250:kworker.0 00010000:00010000:1.0:1788204904.644449:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204905.668578:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204905.988441:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204905.988453:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204905.988458:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204905.988475:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6361 00000100:00080000:2.0:1788204905.988488:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204905.988496:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204905.988546:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062188014848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204905.988578:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204905.988583:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204905.988585:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204905.988590:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6361 00000100:00080000:2.0:1788204905.988595:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204905.988600:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204905.988612:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204905.988630:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062188014976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204905.988643:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204905.988647:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204905.988649:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204905.988654:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6361 00000100:00080000:2.0:1788204905.988659:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788204905.988666:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188014848:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204905.988669:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204905.988683:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204905.988700:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062188015104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204905.988716:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204905.988720:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204905.988722:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204905.988727:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6361 00000100:00080000:2.0:1788204905.988732:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204905.988738:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204905.988744:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204905.988782:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062188015232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204905.988793:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204905.988797:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204905.988800:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204905.988805:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6361 00000100:00080000:2.0:1788204905.988811:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204905.988819:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:1.0:1788204905.988826:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:2.0:1788204905.988831:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204905.988867:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062188015360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204905.988901:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204905/real 1788204905] req@ffff8e82894e6300 x1875062188014848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204905.988930:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062188014848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204905.988941:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204905.988946:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204905.988950:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188014848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204905.988959:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188014976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204905.988985:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204905.989000:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204905/real 1788204905] req@ffff8e82894e4a80 x1875062188014976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204905.989013:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062188014976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204905.989023:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204905.989026:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204905.989029:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188014976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204905.989066:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188015104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204905.989089:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204905.989102:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204905/real 1788204905] req@ffff8e82894e6680 x1875062188015104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204905.989115:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062188015104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204905.989130:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204905.989134:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204905.989138:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188015104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204905.989145:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188015232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204905.989166:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204905.989178:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204905/real 1788204905] req@ffff8e82894e7800 x1875062188015232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204905.989189:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062188015232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204905.989197:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204905.989201:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204905.989204:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188015232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788204905.989211:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188015360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204905.989233:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204905.989245:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204905/real 1788204905] req@ffff8e82894e7100 x1875062188015360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204905.989258:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062188015360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204905.989266:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204905.989269:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204905.989273:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188015360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204906.692480:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204907.716493:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204908.740489:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204909.764442:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204910.788511:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204911.108425:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204911.108435:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204911.108439:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204911.108448:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6366 00000100:00080000:2.0:1788204911.108458:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204911.108465:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204911.108496:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062188015488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204911.108519:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204911.108526:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204911.108528:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204911.108540:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6366 00000100:00080000:2.0:1788204911.108546:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204911.108551:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204911.108561:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188015488:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788204911.108566:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204911.108581:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062188015616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204911.108589:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204911.108592:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204911.108594:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204911.108597:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6366 00000100:00080000:2.0:1788204911.108600:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204911.108603:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204911.108607:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204911.108614:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062188015744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204911.108621:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204911.108623:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204911.108625:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204911.108627:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6366 00000100:00080000:2.0:1788204911.108630:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204911.108634:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:1.0:1788204911.108638:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:2.0:1788204911.108640:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204911.108648:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062188015872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204911.108655:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204911.108658:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204911.108675:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204911.108679:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6366 00000100:00080000:2.0:1788204911.108682:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204911.108684:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204911.108688:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204911.108700:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062188016000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204911.108731:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204911/real 1788204911] req@ffff8e82894e6d80 x1875062188015488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204911.108757:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062188015488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204911.108772:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204911.108779:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204911.108783:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188015488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204911.108808:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188015616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204911.108831:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204911.108841:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204911/real 1788204911] req@ffff8e82894e5c00 x1875062188015616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204911.108851:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062188015616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204911.108859:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204911.108863:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204911.108868:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188015616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204911.108874:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188015744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204911.108894:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204911.108907:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204911/real 1788204911] req@ffff8e82894e4380 x1875062188015744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204911.108920:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062188015744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204911.108926:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204911.108928:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204911.108930:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188015744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204911.108935:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188015872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204911.108950:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204911.108957:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204911/real 1788204911] req@ffff8e82894e5f80 x1875062188015872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204911.108967:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062188015872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204911.108975:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204911.108978:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204911.108981:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188015872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788204911.108988:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188016000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204911.109007:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204911.109020:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204911/real 1788204911] req@ffff8e82894e7100 x1875062188016000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204911.109032:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062188016000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204911.109040:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204911.109043:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204911.109047:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188016000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204911.812458:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204912.836465:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204913.860470:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204914.884457:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204915.908480:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204916.228451:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204916.228460:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204916.228464:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204916.228472:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6371 00000100:00080000:1.0:1788204916.228481:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204916.228487:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204916.228519:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062188016128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204916.228544:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204916.228547:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204916.228548:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204916.228551:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6371 00000100:00080000:1.0:1788204916.228554:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204916.228557:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204916.228564:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204916.228591:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062188016256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204916.228600:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788204916.228603:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188016128:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204916.228604:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204916.228607:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204916.228615:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6371 00000100:00080000:1.0:1788204916.228618:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204916.228622:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204916.228626:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204916.228635:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062188016384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204916.228643:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204916.228645:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204916.228647:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204916.228649:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6371 00000100:00080000:1.0:1788204916.228660:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204916.228663:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204916.228666:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788204916.228671:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204916.228675:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062188016512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204916.228690:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204916.228693:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204916.228695:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204916.228699:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6371 00000100:00080000:1.0:1788204916.228704:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204916.228709:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204916.228713:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204916/real 1788204916] req@ffff8e828d722300 x1875062188016128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788204916.228716:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204916.228734:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062188016128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204916.228738:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062188016640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204916.228751:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204916.228758:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204916.228762:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188016128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204916.228782:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188016256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204916.228796:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204916.228804:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204916/real 1788204916] req@ffff8e828d720380 x1875062188016256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204916.228812:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062188016256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204916.228818:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204916.228820:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204916.228822:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188016256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204916.228826:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188016384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204916.228836:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204916.228843:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204916/real 1788204916] req@ffff8e828d722d80 x1875062188016384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204916.228850:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062188016384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204916.228855:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204916.228856:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204916.228858:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188016384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204916.228862:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188016512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204916.228872:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204916.228878:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204916/real 1788204916] req@ffff8e828d721180 x1875062188016512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204916.228884:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062188016512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204916.228889:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204916.228891:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204916.228893:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188016512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204916.228896:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188016640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204916.228905:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204916.228910:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204916/real 1788204916] req@ffff8e828d720a80 x1875062188016640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204916.228922:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062188016640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204916.228927:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204916.228928:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204916.228930:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188016640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204916.932472:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204917.956468:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204918.980468:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204920.004558:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204921.028551:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204921.348428:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204921.348438:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204921.348442:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204921.348452:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6376 00000100:00080000:1.0:1788204921.348469:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204921.348476:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204921.348524:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062188016768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204921.348550:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204921.348554:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204921.348556:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204921.348560:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6376 00000100:00080000:1.0:1788204921.348564:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204921.348568:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204921.348577:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204921.348589:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062188016896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204921.348595:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188016768:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204921.348605:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204921.348609:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204921.348610:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204921.348615:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6376 00000100:00080000:1.0:1788204921.348618:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204921.348622:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204921.348627:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204921.348638:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062188017024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204921.348648:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204921.348651:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204921.348652:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204921.348656:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6376 00000100:00080000:1.0:1788204921.348659:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204921.348662:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204921.348666:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788204921.348668:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204921.348679:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062188017152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204921.348701:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204921.348704:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204921.348706:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788204921.348710:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204921/real 1788204921] req@ffff8e828d720a80 x1875062188016768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204921.348713:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6376 00000100:00080000:1.0:1788204921.348718:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204921.348726:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204921.348731:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204921.348738:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062188016768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204921.348743:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062188017280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204921.348756:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204921.348762:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204921.348767:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188016768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204921.348798:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188016896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204921.348815:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204921.348826:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204921/real 1788204921] req@ffff8e828d721180 x1875062188016896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204921.348838:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062188016896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204921.348854:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204921.348857:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204921.348860:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188016896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204921.348866:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188017024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204921.348881:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204921.348889:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204921/real 1788204921] req@ffff8e828d722d80 x1875062188017024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204921.348899:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062188017024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204921.348906:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204921.348909:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204921.348911:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188017024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204921.348917:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188017152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204921.348931:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204921.348937:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204921/real 1788204921] req@ffff8e828d720380 x1875062188017152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204921.348944:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062188017152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204921.348948:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204921.348950:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204921.348953:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188017152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204921.348956:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188017280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204921.348965:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204921.348970:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204921/real 1788204921] req@ffff8e828d722300 x1875062188017280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204921.348976:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062188017280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204921.348980:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204921.348981:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204921.348983:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188017280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204922.052511:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204923.076549:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204924.100510:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204925.124546:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204926.148509:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204926.468383:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204926.468393:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204926.468413:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204926.468423:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6381 00000100:00080000:1.0:1788204926.468433:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204926.468440:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204926.468479:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062188017408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204926.468507:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204926.468510:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204926.468511:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204926.468514:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6381 00000100:00080000:1.0:1788204926.468517:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204926.468519:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204926.468528:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204926.468536:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062188017536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204926.468544:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204926.468547:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204926.468549:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204926.468554:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6381 00000100:00080000:1.0:1788204926.468558:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204926.468562:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204926.468567:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204926.468578:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062188017664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204926.468590:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204926.468594:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204926.468598:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788204926.468601:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188017408:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204926.468607:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6381 00000100:00080000:1.0:1788204926.468619:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204926.468627:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204926.468634:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204926.468651:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062188017792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204926.468661:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204926.468664:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204926.468665:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204926.468668:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6381 00000100:00080000:1.0:1788204926.468678:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204926.468682:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204926.468686:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788204926.468694:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204926.468695:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062188017920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204926.468749:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204926/real 1788204926] req@ffff8e828d723800 x1875062188017408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204926.468768:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062188017408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204926.468779:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204926.468783:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204926.468787:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188017408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204926.468796:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188017536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204926.468818:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204926.468830:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204926/real 1788204926] req@ffff8e828d721500 x1875062188017536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204926.468841:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062188017536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204926.468850:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204926.468853:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204926.468856:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188017536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204926.468880:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188017664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204926.468897:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204926.468907:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204926/real 1788204926] req@ffff8e828d720000 x1875062188017664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204926.468919:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062188017664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204926.468927:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204926.468930:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204926.468933:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188017664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204926.468939:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188017792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204926.468955:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204926.468964:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204926/real 1788204926] req@ffff8e828d722680 x1875062188017792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204926.468975:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062188017792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204926.468983:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204926.468987:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204926.468990:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188017792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204926.468996:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188017920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204926.469014:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204926.469034:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204926/real 1788204926] req@ffff8e828d720e00 x1875062188017920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204926.469046:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062188017920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204926.469054:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204926.469058:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204926.469061:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188017920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204927.172475:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204928.196523:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204929.220519:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204930.244483:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204931.268445:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204931.588676:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204931.588685:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204931.588689:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204931.588698:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6386 00000100:00080000:2.0:1788204931.588709:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204931.588715:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204931.588750:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062188018048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204931.588775:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204931.588779:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204931.588781:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204931.588784:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6386 00000100:00080000:2.0:1788204931.588788:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204931.588791:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204931.588800:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204931.588811:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062188018176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204931.588822:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204931.588825:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204931.588827:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788204931.588830:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188018048:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204931.588832:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6386 00000100:00080000:2.0:1788204931.588838:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204931.588850:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204931.588858:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204931.588869:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062188018304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204931.588879:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204931.588882:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204931.588884:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204931.588887:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6386 00000100:00080000:2.0:1788204931.588891:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788204931.588897:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204931.588898:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204931.588906:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204931.588917:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062188018432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204931.588932:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204931.588935:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204931.588937:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204931.588940:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6386 00000100:00080000:2.0:1788204931.588942:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204931.588945:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204931.588952:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204931/real 1788204931] req@ffff8e82894e7800 x1875062188018048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788204931.588954:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204931.588969:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062188018560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204931.588977:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062188018048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204931.588990:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204931.588994:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204931.588999:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188018048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204931.589020:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188018176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204931.589042:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204931.589053:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204931/real 1788204931] req@ffff8e82894e6680 x1875062188018176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204931.589064:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062188018176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204931.589072:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204931.589075:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204931.589079:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188018176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204931.589084:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188018304:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204931.589100:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204931.589109:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204931/real 1788204931] req@ffff8e82894e4a80 x1875062188018304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204931.589119:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062188018304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204931.589127:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204931.589129:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204931.589132:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188018304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204931.589138:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188018432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204931.589152:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204931.589160:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204931/real 1788204931] req@ffff8e82894e6300 x1875062188018432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204931.589171:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062188018432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204931.589179:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204931.589182:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204931.589184:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188018432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204931.589189:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188018560:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204931.589202:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204931.589210:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204931/real 1788204931] req@ffff8e82894e4e00 x1875062188018560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204931.589219:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062188018560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204931.589227:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204931.589230:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204931.589233:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188018560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204932.292475:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204933.316574:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204934.340535:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204935.364486:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204936.388474:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204936.708397:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204936.708408:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204936.708412:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204936.708422:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6391 00000100:00080000:2.0:1788204936.708431:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204936.708440:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204936.708473:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062188018688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204936.708500:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204936.708504:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204936.708506:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204936.708510:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6391 00000100:00080000:2.0:1788204936.708514:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204936.708519:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204936.708531:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204936.708532:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188018688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204936.708549:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062188018816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204936.708563:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204936.708567:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204936.708569:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204936.708573:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6391 00000100:00080000:2.0:1788204936.708576:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204936.708581:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204936.708587:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788204936.708589:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204936.708602:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062188018944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204936.708612:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204936.708616:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204936.708617:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204936.708623:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6391 00000100:00100000:1.0:1788204936.708624:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204936/real 1788204936] req@ffff8e82894e4000 x1875062188018688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204936.708628:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204936.708638:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204936.708643:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062188018688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788204936.708644:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788204936.708657:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204936.708662:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204936.708664:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062188019072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204936.708666:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188018688:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204936.708680:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204936.708683:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204936.708686:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788204936.708689:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188018816:192.168.201.121@tcp:38:kworker.0 00000100:00080000:2.0:1788204936.708691:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6391 00000100:00080000:2.0:1788204936.708698:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204936.708703:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:1.0:1788204936.708708:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:2.0:1788204936.708708:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204936.708718:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204936/real 1788204936] req@ffff8e82894e7480 x1875062188018816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204936.708723:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062188019200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204936.708732:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062188018816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204936.708742:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204936.708744:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204936.708747:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188018816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204936.708751:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188018944:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204936.708763:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204936.708769:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204936/real 1788204936] req@ffff8e82894e6a00 x1875062188018944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204936.708775:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062188018944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204936.708780:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204936.708782:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204936.708783:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188018944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204936.708787:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188019072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204936.708796:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204936.708801:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204936/real 1788204936] req@ffff8e82894e5880 x1875062188019072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204936.708807:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062188019072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204936.708812:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204936.708813:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204936.708815:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188019072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788204936.708822:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188019200:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204936.708831:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204936.708836:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204936/real 1788204936] req@ffff8e82894e4000 x1875062188019200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204936.708842:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062188019200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204936.708846:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204936.708848:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204936.708849:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188019200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204937.412657:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204938.436454:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204939.460464:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204940.484462:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204941.508472:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204941.829362:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204941.829369:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204941.829372:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204941.829379:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6396 00000100:00080000:2.0:1788204941.829387:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204941.829397:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204941.829423:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062188019328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204941.829441:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204941.829444:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204941.829445:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204941.829448:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6396 00000100:00080000:2.0:1788204941.829450:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204941.829457:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204941.829464:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204941.829466:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188019328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204941.829474:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062188019456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204941.829486:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204941.829488:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204941.829490:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204941.829492:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6396 00000100:00080000:2.0:1788204941.829494:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204941.829497:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204941.829499:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204941.829506:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062188019584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204941.829511:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204941.829514:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204941.829515:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204941.829517:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6396 00000100:00080000:2.0:1788204941.829519:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204941.829522:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:0.0:1788204941.829526:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:2.0:1788204941.829528:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204941.829535:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062188019712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204941.829541:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204941.829543:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204941.829544:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204941.829547:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6396 00000100:00080000:2.0:1788204941.829550:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204941.829552:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204941.829556:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204941.829562:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062188019840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204941.829566:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204941/real 1788204941] req@ffff8e82894e4e00 x1875062188019328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204941.829586:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062188019328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204941.829597:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204941.829600:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204941.829602:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188019328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204941.829619:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188019456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204941.829634:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204941.829641:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204941/real 1788204941] req@ffff8e82894e6300 x1875062188019456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204941.829648:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062188019456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204941.829653:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204941.829654:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204941.829656:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188019456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204941.829659:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188019584:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788204941.829668:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204941.829673:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204941/real 1788204941] req@ffff8e82894e4a80 x1875062188019584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204941.829679:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062188019584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204941.829683:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204941.829685:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204941.829686:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188019584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204941.829689:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188019712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204941.829698:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204941.829703:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204941/real 1788204941] req@ffff8e82894e6680 x1875062188019712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204941.829709:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062188019712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204941.829713:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204941.829714:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204941.829716:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188019712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204941.829719:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188019840:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788204941.829727:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204941.829731:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204941/real 1788204941] req@ffff8e82894e7800 x1875062188019840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204941.829737:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062188019840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204941.829740:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204941.829742:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204941.829743:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188019840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204942.532477:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204943.556487:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204944.580470:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204945.604481:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204946.628484:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204946.948435:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204946.948442:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204946.948445:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204946.948453:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6402 00000100:00080000:0.0:1788204946.948461:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204946.948466:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204946.948506:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062188019968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204946.948528:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204946.948531:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204946.948533:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204946.948537:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6402 00000100:00080000:0.0:1788204946.948540:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204946.948544:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204946.948553:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204946.948564:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062188020096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204946.948572:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188019968:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204946.948573:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204946.948575:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204946.948577:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204946.948580:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6402 00000100:00080000:0.0:1788204946.948583:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204946.948586:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204946.948590:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204946.948598:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062188020224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204946.948617:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204946.948620:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204946.948622:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204946.948625:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6402 00000100:00080000:0.0:1788204946.948627:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788204946.948629:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204946.948631:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204946.948634:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204946.948643:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062188020352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204946.948651:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204946.948653:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204946.948654:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204946.948657:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6402 00000100:00080000:0.0:1788204946.948660:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204946.948663:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204946.948666:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204946.948667:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204946/real 1788204946] req@ffff8e828793ca80 x1875062188019968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204946.948674:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062188020480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204946.948682:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062188019968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204946.948691:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204946.948694:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204946.948697:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188019968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204946.948720:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188020096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204946.948738:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204946.948750:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204946/real 1788204946] req@ffff8e828793c000 x1875062188020096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204946.948761:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062188020096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204946.948768:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204946.948771:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204946.948774:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188020096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204946.948780:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188020224:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204946.948794:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204946.948801:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204946/real 1788204946] req@ffff8e828793dc00 x1875062188020224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204946.948808:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062188020224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204946.948813:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204946.948814:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204946.948816:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188020224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204946.948819:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188020352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204946.948830:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204946.948835:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204946/real 1788204946] req@ffff8e828793f480 x1875062188020352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204946.948841:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062188020352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204946.948845:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204946.948847:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204946.948848:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188020352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788204946.948851:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188020480:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204946.948859:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204946.948864:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204946/real 1788204946] req@ffff8e828793c380 x1875062188020480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204946.948874:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062188020480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204946.948882:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204946.948885:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204946.948888:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188020480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204947.652488:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204948.676456:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204949.700527:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204950.724473:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204951.748477:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204952.068452:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204952.068463:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204952.068467:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204952.068477:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6407 00000100:00080000:0.0:1788204952.068490:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204952.068497:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204952.068531:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062188020608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204952.068558:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204952.068561:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204952.068563:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204952.068567:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6407 00000100:00080000:0.0:1788204952.068571:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204952.068575:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204952.068586:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204952.068599:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062188020736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204952.068612:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188020608:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204952.068616:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204952.068628:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204952.068630:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204952.068634:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6407 00000100:00080000:0.0:1788204952.068638:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204952.068642:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204952.068647:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204952.068662:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062188020864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204952.068671:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204952.068675:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204952.068677:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204952.068681:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6407 00000100:00080000:0.0:1788204952.068699:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204952.068704:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204952.068709:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788204952.068713:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788204952.068731:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062188020992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204952.068744:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204952.068747:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204952.068749:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204952.068754:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6407 00000100:00080000:0.0:1788204952.068758:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204952.068762:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204952.068769:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204952.068778:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204952/real 1788204952] req@ffff8e828793d500 x1875062188020608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204952.068786:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062188021120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204952.068805:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062188020608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204952.068815:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204952.068819:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204952.068822:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188020608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204952.068851:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188020736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204952.068874:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204952.068891:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204952/real 1788204952] req@ffff8e828793fb80 x1875062188020736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204952.068900:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062188020736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204952.068906:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204952.068908:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204952.068910:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188020736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204952.068914:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188020864:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204952.068927:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204952.068934:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204952/real 1788204952] req@ffff8e828793d880 x1875062188020864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204952.068941:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062188020864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204952.068945:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204952.068947:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204952.068948:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188020864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204952.068965:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188020992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204952.068980:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204952.068986:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204952/real 1788204952] req@ffff8e828793e300 x1875062188020992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204952.068993:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062188020992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204952.069002:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204952.069004:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204952.069006:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188020992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788204952.069009:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188021120:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204952.069019:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204952.069024:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204952/real 1788204952] req@ffff8e828793c380 x1875062188021120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204952.069030:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062188021120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204952.069035:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204952.069036:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204952.069038:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188021120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204952.772457:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204953.796452:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204954.820485:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204955.844545:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204956.868476:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204957.188438:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204957.188455:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204957.188459:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204957.188469:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6412 00000100:00080000:2.0:1788204957.188480:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204957.188487:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204957.188522:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062188021248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204957.188547:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204957.188551:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204957.188552:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204957.188556:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6412 00000100:00080000:2.0:1788204957.188560:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204957.188564:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204957.188573:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204957.188585:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188021248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788204957.188588:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062188021376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204957.188600:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204957.188605:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204957.188607:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204957.188611:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6412 00000100:00080000:2.0:1788204957.188615:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204957.188619:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204957.188633:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788204957.188638:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788204957.188648:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062188021504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204957.188659:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204957.188662:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204957.188664:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204957.188669:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6412 00000100:00080000:2.0:1788204957.188673:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788204957.188675:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204957/real 1788204957] req@ffff8e82894e7100 x1875062188021248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204957.188680:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204957.188688:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204957.188691:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062188021248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204957.188701:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1788204957.188703:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062188021632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204957.188706:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204957.188711:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188021248:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204957.188723:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204957.188727:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204957.188729:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204957.188734:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6412 00000100:00080000:2.0:1788204957.188737:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204957.188741:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204957.188746:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188021376:192.168.201.121@tcp:38:kworker.0 00000008:00100000:2.0:1788204957.188746:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204957.188762:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062188021760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788204957.188768:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204957.188780:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204957/real 1788204957] req@ffff8e82894e5f80 x1875062188021376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204957.188788:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062188021376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204957.188794:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204957.188797:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204957.188798:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188021376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204957.188802:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188021504:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204957.188812:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204957.188818:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204957/real 1788204957] req@ffff8e82894e4380 x1875062188021504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204957.188824:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062188021504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204957.188831:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204957.188834:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204957.188846:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188021504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204957.188856:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188021632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204957.188907:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204957.188918:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204957/real 1788204957] req@ffff8e82894e5c00 x1875062188021632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204957.188929:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062188021632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204957.188936:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204957.188939:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204957.188942:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188021632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788204957.188957:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188021760:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204957.188974:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204957.188984:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204957/real 1788204957] req@ffff8e82894e6d80 x1875062188021760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204957.188995:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062188021760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204957.189001:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204957.189004:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204957.189008:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188021760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204957.892609:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204958.916463:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204959.940449:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204960.964461:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204961.988514:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204962.308387:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204962.308398:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204962.308403:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204962.308413:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6417 00000100:00080000:2.0:1788204962.308424:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204962.308431:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204962.308466:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062188021888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204962.308497:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204962.308501:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204962.308504:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204962.308508:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6417 00000100:00080000:2.0:1788204962.308512:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204962.308518:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204962.308519:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188021888:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788204962.308528:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204962.308544:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062188022016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204962.308555:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204962.308558:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204962.308560:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204962.308563:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6417 00000100:00080000:2.0:1788204962.308568:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204962.308572:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:1.0:1788204962.308575:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:2.0:1788204962.308579:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204962.308593:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062188022144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204962.308601:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204962.308604:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204962.308606:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788204962.308610:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204962/real 1788204962] req@ffff8e82894e6d80 x1875062188021888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788204962.308613:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6417 00000100:00080000:2.0:1788204962.308620:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204962.308626:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204962.308630:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062188021888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788204962.308633:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788204962.308643:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204962.308646:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788204962.308650:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062188022272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788204962.308651:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188021888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788204962.308670:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188022016:192.168.201.121@tcp:38:kworker.0 00000100:00080000:2.0:1788204962.308672:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204962.308677:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204962.308679:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204962.308684:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6417 00000800:00000100:1.0:1788204962.308688:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788204962.308690:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204962.308696:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204962.308700:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204962/real 1788204962] req@ffff8e82894e5c00 x1875062188022016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788204962.308702:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204962.308714:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062188022016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788204962.308720:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062188022400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204962.308722:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204962.308729:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204962.308736:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188022016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204962.308746:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188022144:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788204962.308772:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204962.308796:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204962/real 1788204962] req@ffff8e82894e4380 x1875062188022144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204962.308804:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062188022144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204962.308809:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204962.308811:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204962.308813:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188022144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788204962.308821:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188022272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204962.308834:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204962.308840:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204962/real 1788204962] req@ffff8e82894e5f80 x1875062188022272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204962.308847:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062188022272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204962.308851:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204962.308853:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204962.308855:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188022272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788204962.308857:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188022400:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788204962.308866:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788204962.308871:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204962/real 1788204962] req@ffff8e82894e7100 x1875062188022400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788204962.308876:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062188022400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788204962.308885:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788204962.308887:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788204962.308889:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188022400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204963.012465:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204964.036600:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204965.060481:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204966.084479:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204967.108474:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788204967.428400:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204967.428408:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788204967.428415:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204967.428423:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6422 00000100:00080000:1.0:1788204967.428432:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204967.428438:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788204967.428468:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720e00 x1875062188022528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204967.428491:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204967.428493:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788204967.428495:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204967.428497:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6422 00000100:00080000:1.0:1788204967.428499:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204967.428501:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204967.428508:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204967.428517:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722680 x1875062188022656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204967.428524:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204967.428526:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788204967.428527:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204967.428530:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6422 00000100:00080000:1.0:1788204967.428531:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788204967.428535:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188022528:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788204967.428536:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204967.428541:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788204967.428547:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720000 x1875062188022784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204967.428553:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204967.428555:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788204967.428557:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204967.428559:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6422 00000100:00080000:1.0:1788204967.428560:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204967.428562:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204967.428565:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204967.428578:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721500 x1875062188022912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788204967.428584:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788204967.428586:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788204967.428587:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788204967.428589:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6422 00000100:00080000:1.0:1788204967.428592:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788204967.428594:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788204967.428597:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788204967.428603:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d723800 x1875062188023040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788204967.428641:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188022656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204967.428682:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188022784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204967.428721:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188022912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204967.428746:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188023040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204968.132463:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204969.156454:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204970.180478:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788204970.500693:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788204970.500749:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788204970.500769:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204970.500820:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204970.500830:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204970.500842:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:3.0:1788204970.500853:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:0.0:1788204970.500881:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204967/real 1788204970] req@ffff8e828d720000 x1875062188022784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204970.500910:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720000 x1875062188022784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204970.500922:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204970.500928:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204970.500932:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188022784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788204970.500940:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204967/real 1788204970] req@ffff8e828d721500 x1875062188022912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204970.500952:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721500 x1875062188022912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204970.500959:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204970.500963:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204970.500967:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188022912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204970.500973:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204967/real 1788204970] req@ffff8e828d723800 x1875062188023040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204970.500984:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d723800 x1875062188023040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204970.500990:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204970.500994:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204970.500997:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d723800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188023040:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788204970.501003:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204967/real 1788204970] req@ffff8e828d720e00 x1875062188022528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204970.501013:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720e00 x1875062188022528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204970.501020:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204970.501023:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204970.501026:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188022528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204970.501031:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204967/real 1788204970] req@ffff8e828d722680 x1875062188022656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204970.501041:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722680 x1875062188022656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204970.501048:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204970.501051:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788204970.501054:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188022656:192.168.201.121@tcp:38:kworker.0 00010000:00010000:1.0:1788204971.204519:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204972.228416:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204972.548431:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204972.548439:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204972.548443:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204972.548454:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6427 00000100:00080000:2.0:1788204972.548464:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204972.548472:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204972.548506:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062188023168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204972.548535:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204972.548539:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204972.548542:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204972.548547:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6427 00000100:00080000:2.0:1788204972.548551:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204972.548557:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204972.548566:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204972.548580:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062188023296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204972.548591:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204972.548594:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204972.548596:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204972.548611:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6427 00000100:00080000:2.0:1788204972.548615:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204972.548619:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204972.548625:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204972.548638:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062188023424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204972.548649:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204972.548652:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204972.548654:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204972.548659:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6427 00000100:00080000:2.0:1788204972.548662:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204972.548669:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788204972.548670:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188023168:192.168.201.121@tcp:250:kworker.0 00000008:00100000:2.0:1788204972.548674:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204972.548686:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062188023552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204972.548696:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204972.548699:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204972.548701:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204972.548704:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6427 00000100:00080000:2.0:1788204972.548708:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204972.548712:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204972.548717:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204972.548728:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062188023680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204972.548745:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204972.548784:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204972/real 1788204972] req@ffff8e82894e7800 x1875062188023168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204972.548801:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062188023168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788204997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204972.548811:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204972.548815:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204972.548819:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188023168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204972.548826:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188023296:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204972.548845:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204972.548856:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204972/real 1788204972] req@ffff8e82894e6680 x1875062188023296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204972.548866:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062188023296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204972.548873:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204972.548876:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204972.548879:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188023296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204972.548884:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188023424:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204972.548900:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204972.548910:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204972/real 1788204972] req@ffff8e82894e4a80 x1875062188023424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204972.548920:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062188023424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788204997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204972.548927:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204972.548930:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204972.548933:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188023424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204972.548970:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188023552:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204972.548990:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204972.548999:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204972/real 1788204972] req@ffff8e82894e6300 x1875062188023552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204972.549011:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062188023552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204972.549017:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204972.549020:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204972.549023:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188023552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204972.549033:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188023680:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204972.549054:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204972.549063:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204972/real 1788204972] req@ffff8e82894e4e00 x1875062188023680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204972.549073:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062188023680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788204997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204972.549079:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204972.549082:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204972.549085:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188023680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204973.252463:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204974.276481:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204975.300483:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204976.324504:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204977.348440:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204977.668389:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204977.668400:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204977.668404:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204977.668413:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6432 00000100:00080000:0.0:1788204977.668422:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204977.668428:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204977.668457:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062188023808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204977.668476:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204977.668479:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204977.668481:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204977.668492:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6432 00000100:00080000:0.0:1788204977.668496:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204977.668500:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188023808:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204977.668504:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204977.668526:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204977.668542:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062188023936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204977.668553:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204977.668556:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204977.668558:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204977.668562:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6432 00000100:00080000:0.0:1788204977.668566:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788204977.668567:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204977.668574:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204977.668579:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204977.668589:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062188024064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204977.668596:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204977.668598:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204977.668599:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204977.668603:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6432 00000100:00080000:0.0:1788204977.668605:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204977.668608:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204977/real 1788204977] req@ffff8e828793f480 x1875062188023808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204977.668611:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204977.668617:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204977.668627:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062188024192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204977.668630:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062188023808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204977.668642:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204977.668651:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204977.668656:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788204977.668659:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788204977.668663:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6432 00000100:00080000:3.0:1788204977.668664:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:0.0:1788204977.668670:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204977.668674:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188023808:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204977.668676:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204977.668683:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204977.668694:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062188024320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204977.668698:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188023936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204977.668721:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204977.668730:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204977/real 1788204977] req@ffff8e828793dc00 x1875062188023936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204977.668737:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062188023936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204977.668742:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204977.668744:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204977.668746:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188023936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204977.668749:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188024064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204977.668759:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204977.668764:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204977/real 1788204977] req@ffff8e828793c000 x1875062188024064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204977.668770:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062188024064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204977.668775:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204977.668776:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204977.668778:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188024064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204977.668782:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188024192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204977.668791:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204977.668796:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204977/real 1788204977] req@ffff8e828793ca80 x1875062188024192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204977.668802:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062188024192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204977.668806:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204977.668808:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204977.668809:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188024192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204977.668817:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188024320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204977.668826:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204977.668831:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204977/real 1788204977] req@ffff8e828793c700 x1875062188024320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204977.668837:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062188024320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204977.668841:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204977.668842:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204977.668844:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188024320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204978.372406:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204979.396466:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204980.420506:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204981.444488:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204982.468479:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204982.788434:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204982.788446:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204982.788450:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204982.788462:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6437 00000100:00080000:0.0:1788204982.788474:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204982.788482:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204982.788518:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062188024448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204982.788546:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204982.788550:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204982.788551:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204982.788556:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6437 00000100:00080000:0.0:1788204982.788559:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204982.788563:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204982.788571:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204982.788585:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062188024576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204982.788596:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188024448:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788204982.788599:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204982.788606:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204982.788608:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204982.788612:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6437 00000100:00080000:0.0:1788204982.788616:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204982.788620:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204982.788624:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204982.788637:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062188024704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204982.788646:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204982.788649:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204982.788651:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204982.788655:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6437 00000100:00080000:0.0:1788204982.788658:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204982.788662:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:3.0:1788204982.788667:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788204982.788670:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204982.788684:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062188024832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204982.788693:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204982.788697:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204982.788699:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204982.788702:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6437 00000100:00080000:0.0:1788204982.788706:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788204982.788711:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204982/real 1788204982] req@ffff8e828793ce00 x1875062188024448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204982.788713:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204982.788721:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204982.788733:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062188024448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204982.788737:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062188024960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204982.788756:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204982.788762:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204982.788767:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188024448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204982.788789:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188024576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204982.788811:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204982.788821:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204982/real 1788204982] req@ffff8e828793f800 x1875062188024576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204982.788829:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062188024576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204982.788835:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204982.788836:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204982.788839:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188024576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204982.788842:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188024704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204982.788856:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204982.788867:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204982/real 1788204982] req@ffff8e828793e680 x1875062188024704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204982.788878:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062188024704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204982.788891:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204982.788894:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204982.788897:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188024704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204982.788902:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188024832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204982.788921:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204982.788930:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204982/real 1788204982] req@ffff8e828793f100 x1875062188024832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204982.788936:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062188024832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204982.788941:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204982.788943:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204982.788945:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188024832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204982.788948:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188024960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204982.788958:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204982.788964:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204982/real 1788204982] req@ffff8e828793c700 x1875062188024960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204982.788970:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062188024960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204982.788974:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204982.788976:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204982.788978:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188024960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204983.492487:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204984.516492:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204985.540524:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204986.564512:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204987.588509:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204987.908475:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204987.908483:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204987.908487:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204987.908495:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6442 00000100:00080000:2.0:1788204987.908504:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204987.908510:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204987.908548:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062188025088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204987.908571:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204987.908574:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204987.908575:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204987.908578:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6442 00000100:00080000:2.0:1788204987.908580:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204987.908583:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204987.908589:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204987.908598:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062188025216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204987.908605:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204987.908608:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204987.908609:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204987.908611:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6442 00000100:00080000:2.0:1788204987.908613:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204987.908616:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204987.908619:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204987.908625:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062188025344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204987.908637:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204987.908640:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204987.908641:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204987.908643:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6442 00000100:00080000:2.0:1788204987.908645:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204987.908647:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204987.908651:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204987.908659:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062188025472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204987.908662:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188025088:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204987.908668:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204987.908678:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204987.908680:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204987.908683:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6442 00000100:00080000:2.0:1788204987.908684:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204987.908686:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204987.908689:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204987.908697:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062188025600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204987.908756:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204987.908814:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204987/real 1788204987] req@ffff8e82894e4000 x1875062188025088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204987.908830:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062188025088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204987.908840:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204987.908843:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204987.908847:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188025088:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204987.908856:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188025216:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204987.908872:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204987.908880:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204987/real 1788204987] req@ffff8e82894e5880 x1875062188025216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204987.908887:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062188025216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204987.908892:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204987.908894:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204987.908896:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188025216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204987.908918:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188025344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204987.908929:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204987.908936:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204987/real 1788204987] req@ffff8e82894e6a00 x1875062188025344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204987.908942:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062188025344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204987.908947:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204987.908949:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204987.908950:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188025344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204987.908954:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188025472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204987.908964:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204987.908970:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204987/real 1788204987] req@ffff8e82894e7480 x1875062188025472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204987.908976:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062188025472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204987.908980:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204987.908982:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204987.908984:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188025472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204987.908986:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188025600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204987.908996:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204987.909001:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204987/real 1788204987] req@ffff8e82894e4e00 x1875062188025600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204987.909006:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062188025600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204987.909011:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204987.909013:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204987.909015:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188025600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204988.612511:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204989.636489:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204990.660513:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204991.684468:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204992.708516:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788204993.028418:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204993.028429:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788204993.028433:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204993.028441:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6448 00000100:00080000:2.0:1788204993.028450:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204993.028456:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788204993.028487:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062188025728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204993.028508:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204993.028510:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788204993.028512:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204993.028514:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6448 00000100:00080000:2.0:1788204993.028516:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204993.028519:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204993.028526:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204993.028533:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062188025856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204993.028539:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204993.028541:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788204993.028543:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204993.028546:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6448 00000100:00080000:2.0:1788204993.028548:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204993.028551:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204993.028554:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788204993.028559:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062188025984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788204993.028564:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204993.028567:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788204993.028568:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204993.028571:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6448 00000100:00080000:2.0:1788204993.028573:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204993.028575:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204993.028577:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204993.028584:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062188026112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788204993.028588:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188025728:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788204993.028591:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788204993.028596:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788204993.028601:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788204993.028607:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6448 00000100:00080000:2.0:1788204993.028608:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788204993.028610:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788204993.028612:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788204993.028619:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062188026240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204993.028672:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204993.028724:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204993/real 1788204993] req@ffff8e82894e6300 x1875062188025728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204993.028741:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062188025728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204993.028751:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204993.028754:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204993.028757:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188025728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204993.028763:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188025856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204993.028777:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204993.028784:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204993/real 1788204993] req@ffff8e82894e4a80 x1875062188025856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204993.028791:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062188025856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204993.028796:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204993.028799:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204993.028800:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188025856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204993.028805:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188025984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204993.028815:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204993.028820:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204993/real 1788204993] req@ffff8e82894e6680 x1875062188025984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204993.028827:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062188025984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204993.028831:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204993.028833:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204993.028834:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188025984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204993.028857:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188026112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204993.028867:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204993.028873:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204993/real 1788204993] req@ffff8e82894e7800 x1875062188026112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204993.028879:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062188026112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204993.028884:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204993.028885:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204993.028887:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188026112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204993.028890:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188026240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204993.028898:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204993.028903:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204993/real 1788204993] req@ffff8e82894e7100 x1875062188026240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204993.028910:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062188026240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204993.028915:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204993.028916:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204993.028918:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188026240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788204993.732520:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204994.756444:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788204995.780487:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204996.804457:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204997.828500:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788204998.148458:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204998.148466:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788204998.148470:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204998.148480:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6453 00000100:00080000:0.0:1788204998.148491:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204998.148497:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788204998.148534:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062188026368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204998.148564:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204998.148567:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788204998.148569:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204998.148573:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6453 00000100:00080000:0.0:1788204998.148577:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204998.148581:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204998.148590:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788204998.148605:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188026368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788204998.148606:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062188026496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204998.148622:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204998.148625:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788204998.148626:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204998.148631:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6453 00000100:00080000:0.0:1788204998.148634:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204998.148639:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204998.148644:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788204998.148656:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062188026624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788204998.148670:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788204998.148678:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788204998.148681:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788204998.148684:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204998.148689:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6453 00000100:00080000:0.0:1788204998.148692:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204998.148695:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204998.148699:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788204998.148708:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062188026752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788204998.148715:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788204998.148716:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204998/real 1788204998] req@ffff8e828793ca80 x1875062188026368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788204998.148721:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788204998.148724:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788204998.148731:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6453 00000100:00080000:0.0:1788204998.148736:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788204998.148738:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788204998.148741:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788204998.148742:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062188026368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788204998.148749:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062188026880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788204998.148753:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204998.148760:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204998.148764:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188026368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788204998.148782:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188026496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204998.148796:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204998.148804:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204998/real 1788204998] req@ffff8e828793c000 x1875062188026496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204998.148810:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062188026496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204998.148819:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204998.148821:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204998.148823:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188026496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204998.148826:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188026624:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788204998.148835:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204998.148850:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204998/real 1788204998] req@ffff8e828793dc00 x1875062188026624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204998.148856:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062188026624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204998.148861:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204998.148863:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204998.148864:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188026624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788204998.148867:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188026752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204998.148882:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204998.148891:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204998/real 1788204998] req@ffff8e828793f480 x1875062188026752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204998.148901:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062188026752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204998.148909:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204998.148912:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204998.148915:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188026752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788204998.148920:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188026880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788204998.148934:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788204998.148941:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788204998/real 1788204998] req@ffff8e828793c380 x1875062188026880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788204998.148947:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062188026880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788204998.148951:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788204998.148953:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788204998.148955:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188026880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788204998.852454:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788204999.876514:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205000.900474:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205001.924510:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205002.948495:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205003.268440:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205003.268450:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205003.268455:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205003.268466:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6458 00000100:00080000:2.0:1788205003.268478:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205003.268485:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788205003.268533:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062188027008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205003.268568:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205003.268572:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205003.268574:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205003.268579:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6458 00000100:00080000:2.0:1788205003.268583:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205003.268588:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205003.268599:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205003.268615:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062188027136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205003.268627:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188027008:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788205003.268636:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205003.268641:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205003.268643:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205003.268647:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6458 00000100:00080000:2.0:1788205003.268652:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205003.268656:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205003.268662:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205003.268675:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062188027264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205003.268684:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205003.268688:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205003.268690:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205003.268694:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6458 00000100:00080000:2.0:1788205003.268699:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788205003.268703:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788205003.268707:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205003.268714:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788205003.268726:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062188027392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205003.268735:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205003.268739:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205003.268741:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205003.268744:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6458 00000100:00080000:2.0:1788205003.268747:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205003.268751:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205003.268756:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788205003.268762:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205003/real 1788205003] req@ffff8e82894e5f80 x1875062188027008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205003.268771:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062188027520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205003.268781:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062188027008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205003.268791:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205003.268795:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205003.268798:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188027008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788205003.268819:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188027136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788205003.268835:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205003.268843:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205003/real 1788205003] req@ffff8e82894e4380 x1875062188027136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205003.268851:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062188027136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205003.268856:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205003.268858:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205003.268860:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188027136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788205003.268863:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188027264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788205003.268873:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205003.268879:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205003/real 1788205003] req@ffff8e82894e5c00 x1875062188027264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205003.268886:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062188027264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205003.268890:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205003.268892:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205003.268893:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188027264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788205003.268896:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188027392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788205003.268907:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205003.268912:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205003/real 1788205003] req@ffff8e82894e6d80 x1875062188027392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205003.268918:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062188027392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205003.268923:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205003.268925:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205003.268927:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188027392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788205003.268930:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188027520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788205003.268938:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205003.268943:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205003/real 1788205003] req@ffff8e82894e7100 x1875062188027520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205003.268949:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062188027520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205003.268954:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205003.268955:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205003.268957:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188027520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788205003.972572:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205004.996525:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205006.020491:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205007.044504:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205008.068552:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205008.388417:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205008.388427:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205008.388431:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205008.388443:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6463 00000100:00080000:2.0:1788205008.388454:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205008.388462:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788205008.388509:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062188027648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205008.388536:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205008.388542:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205008.388543:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205008.388547:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6463 00000100:00080000:2.0:1788205008.388551:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205008.388554:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205008.388564:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205008.388581:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062188027776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205008.388589:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205008.388592:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205008.388594:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205008.388599:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6463 00000100:00080000:2.0:1788205008.388601:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205008.388604:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205008.388607:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205008.388619:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062188027904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205008.388628:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188027648:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788205008.388629:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205008.388636:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205008.388638:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205008.388648:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6463 00000100:00080000:2.0:1788205008.388652:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205008.388655:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205008.388661:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788205008.388673:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062188028032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205008.388681:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205008.388683:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205008.388685:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205008.388691:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6463 00000100:00080000:2.0:1788205008.388696:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205008.388700:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205008.388703:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788205008.388712:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062188028160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205008.388729:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205008.388776:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205008/real 1788205008] req@ffff8e82894e7800 x1875062188027648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205008.388795:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062188027648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205008.388806:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205008.388810:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205008.388815:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188027648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788205008.388822:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188027776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788205008.388844:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205008.388857:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205008/real 1788205008] req@ffff8e82894e6680 x1875062188027776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205008.388869:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062188027776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205008.388878:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205008.388881:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205008.388886:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188027776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788205008.388912:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188027904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788205008.388934:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205008.388952:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205008/real 1788205008] req@ffff8e82894e4a80 x1875062188027904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205008.388970:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062188027904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205008.388991:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205008.388996:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205008.389003:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188027904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788205008.389011:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188028032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788205008.389039:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205008.389063:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205008/real 1788205008] req@ffff8e82894e6300 x1875062188028032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205008.389076:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062188028032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205008.389084:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205008.389087:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205008.389090:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188028032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788205008.389096:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188028160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788205008.389114:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205008.389124:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205008/real 1788205008] req@ffff8e82894e4e00 x1875062188028160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205008.389140:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062188028160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205008.389154:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205008.389157:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205008.389159:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188028160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788205009.092502:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205010.116476:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205011.140446:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205012.164476:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205013.188470:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205013.508392:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205013.508401:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205013.508405:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205013.508416:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6468 00000100:00080000:2.0:1788205013.508426:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205013.508433:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788205013.508467:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062188028288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205013.508496:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205013.508500:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205013.508501:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205013.508505:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6468 00000100:00080000:2.0:1788205013.508510:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205013.508514:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205013.508523:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205013.508537:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062188028416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205013.508549:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205013.508552:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205013.508555:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788205013.508557:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188028288:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788205013.508561:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6468 00000100:00080000:2.0:1788205013.508567:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205013.508571:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205013.508576:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205013.508587:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062188028544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205013.508597:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205013.508600:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205013.508601:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205013.508603:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6468 00000100:00080000:2.0:1788205013.508606:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205013.508609:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205013.508613:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788205013.508621:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062188028672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205013.508628:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788205013.508635:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205013.508637:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205013.508638:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205013.508643:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6468 00000100:00080000:2.0:1788205013.508647:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205013.508651:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205013.508655:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788205013.508668:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062188028800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205013.508679:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205013/real 1788205013] req@ffff8e82894e7480 x1875062188028288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205013.508700:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062188028288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205013.508709:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205013.508713:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205013.508717:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188028288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788205013.508738:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188028416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788205013.508757:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205013.508769:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205013/real 1788205013] req@ffff8e82894e6a00 x1875062188028416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205013.508780:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062188028416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205013.508794:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205013.508797:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205013.508800:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188028416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788205013.508806:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188028544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788205013.508824:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205013.508834:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205013/real 1788205013] req@ffff8e82894e5880 x1875062188028544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205013.508845:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062188028544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205013.508853:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205013.508856:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205013.508859:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188028544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788205013.508865:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188028672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788205013.508883:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205013.508906:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205013/real 1788205013] req@ffff8e82894e4000 x1875062188028672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205013.508917:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062188028672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205013.508925:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205013.508928:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205013.508932:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188028672:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788205013.508938:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188028800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788205013.508957:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205013.508966:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205013/real 1788205013] req@ffff8e82894e4e00 x1875062188028800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205013.508977:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062188028800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205013.508984:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205013.508987:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205013.508989:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188028800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788205014.212460:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205015.236473:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205016.260433:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205017.284466:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205018.308506:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205018.628437:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205018.628444:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205018.628447:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205018.628454:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6473 00000100:00080000:1.0:1788205018.628463:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205018.628468:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788205018.628497:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062188028928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205018.628517:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205018.628519:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205018.628521:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205018.628523:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6473 00000100:00080000:1.0:1788205018.628525:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205018.628527:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788205018.628534:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205018.628540:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062188029056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205018.628547:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205018.628549:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788205018.628550:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205018.628552:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6473 00000100:00080000:1.0:1788205018.628553:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205018.628555:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788205018.628559:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205018.628565:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062188029184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205018.628570:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205018.628572:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788205018.628573:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205018.628575:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6473 00000100:00080000:1.0:1788205018.628577:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205018.628580:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788205018.628582:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788205018.628586:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062188029312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205018.628594:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205018.628597:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788205018.628601:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788205018.628605:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188028928:192.168.201.121@tcp:250:kworker.0 00000100:00080000:1.0:1788205018.628605:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6473 00000100:00080000:1.0:1788205018.628614:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205018.628621:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788205018.628624:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788205018.628629:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062188029440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205018.628700:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205018.628753:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205018/real 1788205018] req@ffff8e828d722300 x1875062188028928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205018.628776:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062188028928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205018.628789:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205018.628794:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205018.628798:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188028928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788205018.628840:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188029056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788205018.628864:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205018.628878:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205018/real 1788205018] req@ffff8e828d720380 x1875062188029056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205018.628891:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062188029056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205018.628899:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205018.628903:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205018.628906:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188029056:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788205018.628912:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188029184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788205018.628933:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205018.628944:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205018/real 1788205018] req@ffff8e828d722d80 x1875062188029184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205018.628956:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062188029184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205018.628963:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205018.628967:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205018.628970:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188029184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788205018.629001:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188029312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788205018.629023:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205018.629035:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205018/real 1788205018] req@ffff8e828d721180 x1875062188029312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205018.629047:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062188029312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205018.629054:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205018.629057:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205018.629060:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188029312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788205018.629065:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188029440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788205018.629083:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205018.629093:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205018/real 1788205018] req@ffff8e828d720a80 x1875062188029440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205018.629104:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062188029440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205018.629112:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205018.629115:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205018.629118:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188029440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788205019.332516:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205020.356518:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205021.380460:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205022.404453:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205023.428483:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788205023.748409:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205023.748418:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788205023.748421:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205023.748430:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6478 00000100:00080000:1.0:1788205023.748439:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205023.748445:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788205023.748475:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720a80 x1875062188029568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205023.748493:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205023.748496:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788205023.748497:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205023.748499:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6478 00000100:00080000:1.0:1788205023.748501:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205023.748504:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788205023.748510:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205023.748517:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d721180 x1875062188029696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205023.748523:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205023.748526:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788205023.748527:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205023.748530:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6478 00000100:00080000:1.0:1788205023.748538:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205023.748541:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788205023.748544:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205023.748551:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188029568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788205023.748552:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722d80 x1875062188029824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205023.748565:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205023.748567:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788205023.748568:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205023.748570:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6478 00000100:00080000:1.0:1788205023.748572:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205023.748574:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788205023.748577:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788205023.748584:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d720380 x1875062188029952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205023.748589:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788205023.748591:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788205023.748592:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205023.748594:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6478 00000100:00080000:1.0:1788205023.748606:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788205023.748609:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:1.0:1788205023.748612:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788205023.748618:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205023.748619:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828d722300 x1875062188030080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788205023.748671:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205023/real 1788205023] req@ffff8e828d720a80 x1875062188029568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205023.748695:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720a80 x1875062188029568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205023.748705:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205023.748708:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205023.748724:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188029568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788205023.748732:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188029696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788205023.748754:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205023.748767:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205023/real 1788205023] req@ffff8e828d721180 x1875062188029696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205023.748778:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d721180 x1875062188029696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205023.748786:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205023.748789:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205023.748793:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d721180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188029696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788205023.748818:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188029824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788205023.748837:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205023.748848:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205023/real 1788205023] req@ffff8e828d722d80 x1875062188029824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205023.748858:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722d80 x1875062188029824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205023.748872:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205023.748875:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205023.748878:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188029824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788205023.748882:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188029952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788205023.748895:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205023.748901:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205023/real 1788205023] req@ffff8e828d720380 x1875062188029952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205023.748908:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d720380 x1875062188029952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205023.748912:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205023.748915:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205023.748918:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d720380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188029952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788205023.748923:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188030080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788205023.748940:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205023.748949:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205023/real 1788205023] req@ffff8e828d722300 x1875062188030080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205023.748960:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828d722300 x1875062188030080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205023.748967:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205023.748970:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205023.748974:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828d722300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188030080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788205024.452475:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205025.476470:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205026.500468:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205027.524524:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205028.548452:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205028.868469:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205028.868479:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205028.868483:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205028.868493:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6483 00000100:00080000:0.0:1788205028.868503:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205028.868516:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788205028.868563:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062188030208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205028.868605:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205028.868609:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205028.868611:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205028.868615:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6483 00000100:00080000:0.0:1788205028.868619:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205028.868635:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205028.868645:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205028.868669:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062188030336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205028.868679:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205028.868682:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205028.868684:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205028.868687:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6483 00000100:00080000:0.0:1788205028.868691:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205028.868695:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205028.868700:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205028.868712:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062188030464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205028.868723:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788205028.868723:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188030208:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788205028.868736:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205028.868738:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205028.868742:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6483 00000100:00080000:0.0:1788205028.868746:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205028.868750:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205028.868755:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205028.868768:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062188030592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205028.868777:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205028.868780:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205028.868782:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205028.868785:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6483 00000100:00080000:0.0:1788205028.868790:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788205028.868791:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788205028.868797:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205028.868802:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205028.868816:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062188030720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205028.868839:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205028/real 1788205028] req@ffff8e828793e300 x1875062188030208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205028.868858:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062188030208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205028.868868:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205028.868872:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205028.868875:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188030208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788205028.868883:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188030336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788205028.868905:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205028.868917:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205028/real 1788205028] req@ffff8e828793d880 x1875062188030336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205028.868929:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062188030336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205028.868937:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205028.868940:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205028.868944:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188030336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788205028.868966:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188030464:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788205028.868986:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205028.869704:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205028/real 1788205028] req@ffff8e828793fb80 x1875062188030464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205028.869716:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062188030464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205028.869724:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205028.869728:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205028.869731:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188030464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788205028.869738:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188030592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788205028.869759:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205028.869770:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205028/real 1788205028] req@ffff8e828793d500 x1875062188030592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205028.869780:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062188030592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205028.869787:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205028.869789:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205028.869792:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188030592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788205028.869798:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188030720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788205028.869814:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205028.869821:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205028/real 1788205028] req@ffff8e828793c380 x1875062188030720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205028.869827:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062188030720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205028.869832:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205028.869834:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205028.869836:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188030720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788205029.572508:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205030.596485:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205031.620474:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205032.644495:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205033.668436:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205033.988427:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205033.988438:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205033.988443:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205033.988453:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6489 00000100:00080000:2.0:1788205033.988463:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205033.988471:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788205033.988503:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062188030848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205033.988523:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205033.988526:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205033.988528:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205033.988532:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6489 00000100:00080000:2.0:1788205033.988549:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205033.988553:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205033.988564:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205033.988576:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062188030976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205033.988585:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788205033.988587:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188030848:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788205033.988588:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205033.988590:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205033.988593:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6489 00000100:00080000:2.0:1788205033.988596:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205033.988600:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205033.988603:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205033.988611:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062188031104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205033.988618:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205033.988620:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205033.988622:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205033.988624:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6489 00000100:00080000:2.0:1788205033.988633:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205033.988636:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205033.988640:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788205033.988649:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062188031232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205033.988656:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205033.988659:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205033.988661:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205033.988663:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6489 00000100:00080000:2.0:1788205033.988666:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205033.988669:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205033.988673:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788205033.988681:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062188031360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205033.988688:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188030976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788205033.988713:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188031104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788205033.988735:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188031232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788205033.988758:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188031360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788205034.692519:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205035.716438:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205036.742733:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1788205037.060714:0:2390:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:1.0:1788205037.060779:0:2390:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788205037.060799:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788205037.060857:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788205037.060867:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788205037.060876:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:1.0:1788205037.060884:0:2390:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:3.0:1788205037.061457:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205033/real 1788205037] req@ffff8e82894e7100 x1875062188031360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205037.061497:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062188031360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205037.061510:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205037.061516:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205037.061521:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188031360:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788205037.061529:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205033/real 1788205037] req@ffff8e82894e4a80 x1875062188030976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205037.061544:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062188030976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205037.061552:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205037.061556:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205037.061558:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188030976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788205037.061565:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205033/real 1788205037] req@ffff8e82894e6680 x1875062188031104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205037.061575:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062188031104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205037.061581:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205037.061584:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205037.061587:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188031104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788205037.061595:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205033/real 1788205037] req@ffff8e82894e7800 x1875062188031232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205037.061605:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062188031232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205037.061611:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205037.061614:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205037.061617:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188031232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788205037.061623:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205033/real 1788205037] req@ffff8e82894e6300 x1875062188030848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205037.061633:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062188030848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205037.061641:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205037.061644:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205037.061647:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188030848:192.168.201.121@tcp:250:kworker.0 00010000:00010000:0.0:1788205037.764516:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205038.788495:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205039.108422:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205039.108432:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205039.108437:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205039.108446:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6494 00000100:00080000:2.0:1788205039.108470:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205039.108477:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788205039.108509:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6d80 x1875062188031488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205039.108543:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205039.108547:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205039.108549:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205039.108554:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6494 00000100:00080000:2.0:1788205039.108558:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205039.108562:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205039.108571:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205039.108589:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5c00 x1875062188031616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205039.108602:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188031488:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788205039.108608:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205039.108613:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205039.108614:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205039.108618:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6494 00000100:00080000:2.0:1788205039.108623:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205039.108626:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205039.108632:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205039.108644:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4380 x1875062188031744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205039.108653:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205039.108656:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205039.108658:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205039.108661:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6494 00000100:00080000:2.0:1788205039.108665:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205039.108668:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205039.108673:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788205039.108679:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:2.0:1788205039.108685:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5f80 x1875062188031872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205039.108695:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205039.108699:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205039.108701:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205039.108704:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6494 00000100:00080000:2.0:1788205039.108708:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205039.108713:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205039.108718:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788205039.108721:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205039/real 1788205039] req@ffff8e82894e6d80 x1875062188031488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788205039.108732:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062188032000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205039.108743:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6d80 x1875062188031488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205039.108755:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205039.108760:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205039.108764:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188031488:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788205039.108789:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188031616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788205039.108811:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205039.108823:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205039/real 1788205039] req@ffff8e82894e5c00 x1875062188031616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205039.108836:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5c00 x1875062188031616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205039.108843:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205039.108846:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205039.108849:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188031616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788205039.108855:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188031744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788205039.108874:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205039.108886:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205039/real 1788205039] req@ffff8e82894e4380 x1875062188031744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205039.108897:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4380 x1875062188031744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205039.108905:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205039.108907:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205039.108910:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188031744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788205039.108916:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188031872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788205039.108936:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205039.108948:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205039/real 1788205039] req@ffff8e82894e5f80 x1875062188031872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205039.108960:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5f80 x1875062188031872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205039.108968:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205039.108971:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205039.108974:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188031872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788205039.108980:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188032000:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788205039.108998:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205039.109015:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205039/real 1788205039] req@ffff8e82894e6300 x1875062188032000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205039.109027:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062188032000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205039.109035:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205039.109037:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205039.109041:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188032000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788205039.812481:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205040.836981:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205041.860474:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205042.885118:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205043.908480:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205044.228469:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205044.228478:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205044.228482:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205044.228493:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6499 00000100:00080000:2.0:1788205044.228503:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205044.228511:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788205044.228547:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062188032128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205044.228572:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205044.228576:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205044.228578:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205044.228582:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6499 00000100:00080000:2.0:1788205044.228586:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205044.228590:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205044.228600:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205044.228604:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188032128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788205044.228616:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062188032256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205044.228627:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205044.228634:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205044.228636:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205044.228640:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6499 00000100:00080000:2.0:1788205044.228644:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205044.228649:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205044.228654:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205044.228666:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062188032384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205044.228669:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788205044.228679:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205044.228683:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205044.228685:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205044.228690:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6499 00000100:00080000:2.0:1788205044.228694:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205044.228698:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205044.228704:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788205044.228716:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062188032512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205044.228718:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205044/real 1788205044] req@ffff8e82894e7800 x1875062188032128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205044.228734:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205044.228738:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205044.228740:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205044.228744:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6499 00000100:00080000:2.0:1788205044.228748:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788205044.228749:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062188032128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205044.228755:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205044.228763:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788205044.228767:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205044.228772:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:2.0:1788205044.228776:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062188032640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205044.228778:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188032128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788205044.228856:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188032256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205044.228874:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205044.228883:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205044/real 1788205044] req@ffff8e82894e6680 x1875062188032256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205044.228896:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062188032256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205044.228901:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205044.228903:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205044.228906:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188032256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205044.228910:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188032384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205044.228921:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205044.228927:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205044/real 1788205044] req@ffff8e82894e4a80 x1875062188032384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205044.228933:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062188032384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205044.228938:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205044.228940:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205044.228942:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188032384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205044.228946:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188032512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205044.228955:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205044.228961:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205044/real 1788205044] req@ffff8e82894e7100 x1875062188032512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205044.228968:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062188032512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205044.228977:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205044.228979:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205044.228981:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188032512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205044.228984:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188032640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205044.228994:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205044.229004:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205044/real 1788205044] req@ffff8e82894e4e00 x1875062188032640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205044.229015:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062188032640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205044.229022:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205044.229025:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205044.229028:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188032640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788205044.932550:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205045.956508:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205046.980531:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205048.004504:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205049.028518:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205049.348451:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205049.348463:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205049.348468:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205049.348480:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6504 00000100:00080000:0.0:1788205049.348493:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205049.348501:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788205049.348548:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062188032768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205049.348586:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205049.348590:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205049.348592:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205049.348597:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6504 00000100:00080000:0.0:1788205049.348601:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205049.348606:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205049.348618:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205049.348627:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188032768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788205049.348633:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062188032896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205049.348645:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205049.348649:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205049.348650:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205049.348654:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6504 00000100:00080000:0.0:1788205049.348657:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205049.348661:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205049.348666:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205049.348677:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062188033024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205049.348686:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205049.348689:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205049.348691:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205049.348695:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6504 00000100:00080000:0.0:1788205049.348698:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205049.348703:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000800:00000100:1.0:1788205049.348704:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000008:00100000:0.0:1788205049.348711:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205049.348722:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062188033152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205049.348731:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205049.348734:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205049.348736:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205049.348747:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6504 00000100:00080000:0.0:1788205049.348753:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788205049.348756:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205049/real 1788205049] req@ffff8e828793f480 x1875062188032768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205049.348760:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205049.348769:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205049.348781:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062188033280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205049.348782:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062188032768/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205049.348818:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205049.348822:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205049.348828:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188032768:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788205049.348851:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188032896:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205049.348883:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205049.348896:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205049/real 1788205049] req@ffff8e828793dc00 x1875062188032896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205049.348909:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062188032896/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205049.348915:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205049.348917:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205049.348919:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188032896:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205049.348923:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188033024:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205049.348937:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205049.348944:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205049/real 1788205049] req@ffff8e828793c000 x1875062188033024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205049.348950:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062188033024/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205049.348955:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205049.348957:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205049.348959:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188033024:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205049.348963:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188033152:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205049.348973:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205049.348979:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205049/real 1788205049] req@ffff8e828793ca80 x1875062188033152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205049.348986:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062188033152/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205049.348990:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205049.348992:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205049.348993:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188033152:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205049.348996:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188033280:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205049.349004:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205049.349009:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205049/real 1788205049] req@ffff8e828793c700 x1875062188033280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205049.349015:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062188033280/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205049.349019:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205049.349020:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205049.349022:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188033280:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788205050.052460:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205051.076560:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205052.100563:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205053.124528:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205054.148577:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205054.468449:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205054.468460:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205054.468465:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205054.468475:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6509 00000100:00080000:0.0:1788205054.468486:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205054.468493:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788205054.468532:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062188033408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205054.468559:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205054.468563:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205054.468564:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205054.468569:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6509 00000100:00080000:0.0:1788205054.468574:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205054.468579:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205054.468589:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205054.468605:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188033408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788205054.468606:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062188033536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205054.468629:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205054.468632:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205054.468634:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205054.468638:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6509 00000100:00080000:0.0:1788205054.468641:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205054.468644:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205054.468650:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205054.468663:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062188033664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205054.468671:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205054.468675:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205054.468677:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205054.468681:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6509 00000100:00080000:0.0:1788205054.468694:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205054.468698:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205054.468704:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788205054.468704:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205054.468722:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062188033792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205054.468734:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205054.468738:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205054.468740:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205054.468744:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6509 00000100:00080000:0.0:1788205054.468749:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205054.468753:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205054.468760:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788205054.468763:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205054/real 1788205054] req@ffff8e828793f100 x1875062188033408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205054.468789:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062188033408/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205054.468791:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062188033920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788205054.468814:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205054.468821:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205054.468825:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188033408:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788205054.468860:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188033536:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205054.468878:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205054.468887:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205054/real 1788205054] req@ffff8e828793e680 x1875062188033536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205054.468895:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062188033536/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205054.468900:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205054.468903:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205054.468905:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188033536:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205054.468908:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188033664:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205054.468919:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205054.468925:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205054/real 1788205054] req@ffff8e828793f800 x1875062188033664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205054.468935:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062188033664/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205054.468940:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205054.468941:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205054.468943:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188033664:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205054.468946:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188033792:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205054.468958:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205054.468967:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205054/real 1788205054] req@ffff8e828793ce00 x1875062188033792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205054.468977:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062188033792/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205054.468985:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205054.468988:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205054.468992:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188033792:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205054.468998:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188033920:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205054.469025:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205054.469037:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205054/real 1788205054] req@ffff8e828793c700 x1875062188033920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205054.469048:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062188033920/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205054.469059:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205054.469061:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205054.469063:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188033920:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788205055.172625:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205056.196749:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205057.221354:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205058.244779:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205059.268698:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205059.588946:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205059.588957:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205059.588961:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205059.588971:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6514 00000100:00080000:3.0:1788205059.588982:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205059.588989:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788205059.589025:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062188034048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205059.589055:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205059.589059:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205059.589061:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205059.589065:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6514 00000100:00080000:3.0:1788205059.589070:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205059.589074:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205059.589083:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205059.589111:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062188034176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205059.589116:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188034048:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788205059.589129:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205059.589132:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788205059.589134:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205059.589139:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6514 00000100:00080000:3.0:1788205059.589143:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205059.589147:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205059.589154:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205059.589166:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062188034304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205059.589175:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205059.589178:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1788205059.589182:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788205059.589183:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205059.589191:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6514 00000100:00080000:3.0:1788205059.589196:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205059.589201:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205059.589206:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788205059.589220:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062188034432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205059.589229:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205059/real 1788205059] req@ffff8e828953bb80 x1875062188034048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205059.589231:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205059.589240:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788205059.589245:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205059.589252:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6514 00000100:00100000:1.0:1788205059.589255:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062188034048/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205059.589257:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205059.589266:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00080000:1.0:1788205059.589269:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788205059.589272:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788205059.589276:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205059.589280:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188034048:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788205059.589288:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062188034560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205059.589302:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188034176:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205059.589347:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205059.589358:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205059/real 1788205059] req@ffff8e8289538a80 x1875062188034176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205059.589368:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062188034176/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205059.589379:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205059.589382:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205059.589385:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188034176:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205059.589389:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188034304:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205059.589401:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205059.589408:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205059/real 1788205059] req@ffff8e828953b100 x1875062188034304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205059.589416:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062188034304/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205059.589422:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205059.589424:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205059.589426:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188034304:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205059.589431:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188034432:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205059.589450:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205059.589461:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205059/real 1788205059] req@ffff8e8289539880 x1875062188034432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205059.589473:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062188034432/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205059.589480:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205059.589482:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205059.589485:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188034432:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205059.589490:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188034560:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205059.589506:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205059.589516:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205059/real 1788205059] req@ffff8e8289539500 x1875062188034560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205059.589526:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062188034560/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205059.589533:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205059.589536:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205059.589539:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188034560:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788205060.292778:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205061.316575:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205062.340701:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205063.364714:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205064.388586:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205064.708420:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205064.708428:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205064.708432:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205064.708440:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6519 00000100:00080000:3.0:1788205064.708448:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205064.708454:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788205064.708484:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062188034688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205064.708508:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205064.708511:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205064.708512:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205064.708515:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6519 00000100:00080000:3.0:1788205064.708517:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205064.708519:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205064.708526:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205064.708540:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062188034816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205064.708546:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205064.708547:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788205064.708548:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205064.708550:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6519 00000100:00080000:3.0:1788205064.708552:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205064.708554:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205064.708556:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205064.708561:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062188034944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205064.708569:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205064.708572:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788205064.708574:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205064.708577:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6519 00000100:00080000:3.0:1788205064.708580:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205064.708585:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205064.708590:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788205064.708600:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062188035072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205064.708609:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205064.708612:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788205064.708614:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205064.708617:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6519 00000100:00080000:3.0:1788205064.708620:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205064.708622:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205064.708625:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788205064.708632:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062188035200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205064.709105:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188034688:192.168.201.121@tcp:250:kworker.0 00000800:00000100:1.0:1788205064.709395:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205064.709439:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205064/real 1788205064] req@ffff8e828953a680 x1875062188034688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205064.709454:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062188034688/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205064.709462:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205064.709466:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205064.709469:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188034688:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788205064.709475:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188034816:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205064.709498:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205064.709507:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205064/real 1788205064] req@ffff8e8289539f80 x1875062188034816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205064.709516:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062188034816/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205064.709522:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205064.709524:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205064.709527:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188034816:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205064.709532:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188034944:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205064.709543:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205064.709548:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205064/real 1788205064] req@ffff8e8289538380 x1875062188034944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205064.709555:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062188034944/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205064.709560:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205064.709561:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205064.709563:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188034944:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205064.709566:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188035072:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205064.709577:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205064.709582:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205064/real 1788205064] req@ffff8e8289539500 x1875062188035072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205064.709589:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062188035072/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205064.709594:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205064.709596:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205064.709597:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188035072:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205064.709600:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188035200:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205064.709611:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205064.709617:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205064/real 1788205064] req@ffff8e8289539880 x1875062188035200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205064.709626:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062188035200/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205064.709634:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205064.709641:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205064.709644:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188035200:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788205065.412481:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205066.437122:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205067.460559:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205068.484525:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205069.509095:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205069.828445:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205069.828453:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205069.828456:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205069.828465:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6524 00000100:00080000:0.0:1788205069.828473:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205069.828478:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788205069.828506:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062188035328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205069.828531:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205069.828533:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205069.828534:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205069.828537:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6524 00000100:00080000:0.0:1788205069.828539:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205069.828541:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205069.828547:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205069.828554:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062188035456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205069.828561:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205069.828564:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205069.828565:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205069.828567:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6524 00000100:00080000:0.0:1788205069.828569:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205069.828572:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205069.828574:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205069.828580:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062188035584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205069.828585:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205069.828588:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205069.828590:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788205069.828590:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188035328:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788205069.828594:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6524 00000100:00080000:0.0:1788205069.828598:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205069.828601:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205069.828604:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205069.828610:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062188035712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205069.828615:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205069.828617:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205069.828619:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205069.828622:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6524 00000100:00080000:0.0:1788205069.828626:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205069.828629:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205069.828634:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205069.828645:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062188035840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205069.828661:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205069.828703:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205069/real 1788205069] req@ffff8e828793ca80 x1875062188035328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205069.828729:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062188035328/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205069.828739:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205069.828743:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205069.828746:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188035328:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788205069.828752:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188035456:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205069.828773:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205069.828786:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205069/real 1788205069] req@ffff8e828793c000 x1875062188035456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205069.828798:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062188035456/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205069.828805:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205069.828808:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205069.828812:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188035456:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205069.828834:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188035584:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205069.828854:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205069.828865:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205069/real 1788205069] req@ffff8e828793dc00 x1875062188035584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205069.828883:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062188035584/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205069.828896:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205069.828899:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205069.828901:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188035584:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205069.828907:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188035712:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205069.828950:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205069.828960:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205069/real 1788205069] req@ffff8e828793f480 x1875062188035712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205069.828971:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062188035712/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205069.828978:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205069.828981:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205069.828984:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188035712:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205069.828989:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188035840:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205069.829004:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205069.829012:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205069/real 1788205069] req@ffff8e828793c380 x1875062188035840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205069.829022:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062188035840/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205069.829029:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205069.829031:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205069.829034:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188035840:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788205070.532556:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205071.556555:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205072.580667:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205073.604519:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205074.628553:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205074.948473:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205074.948483:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205074.948488:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205074.948499:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6530 00000100:00080000:2.0:1788205074.948510:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205074.948517:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788205074.948557:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4000 x1875062188035968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205074.948580:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205074.948583:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205074.948584:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205074.948587:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6530 00000100:00080000:2.0:1788205074.948589:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205074.948592:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205074.948613:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205074.948628:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188035968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:2.0:1788205074.948629:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e5880 x1875062188036096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205074.948652:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205074.948656:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205074.948659:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205074.948663:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6530 00000100:00080000:2.0:1788205074.948669:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205074.948673:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205074.948679:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205074.948690:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6a00 x1875062188036224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205074.948697:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788205074.948704:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:2.0:1788205074.948705:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205074.948708:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205074.948715:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6530 00000100:00080000:2.0:1788205074.948719:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205074.948721:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205074.948725:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788205074.948736:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7480 x1875062188036352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205074.948743:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205074.948746:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205074.948749:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788205074.948750:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205074/real 1788205074] req@ffff8e82894e4000 x1875062188035968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788205074.948753:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6530 00000100:00080000:2.0:1788205074.948757:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205074.948763:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205074.948770:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788205074.948779:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4e00 x1875062188036480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205074.948796:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4000 x1875062188035968/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205074.948808:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205074.948813:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205074.948817:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188035968:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788205074.948868:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188036096:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205074.948890:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205074.948901:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205074/real 1788205074] req@ffff8e82894e5880 x1875062188036096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205074.948913:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e5880 x1875062188036096/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205074.948919:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205074.948921:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205074.948923:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188036096:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205074.948929:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188036224:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205074.948941:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205074.948947:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205074/real 1788205074] req@ffff8e82894e6a00 x1875062188036224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205074.948953:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6a00 x1875062188036224/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205074.948958:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205074.948959:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205074.948961:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188036224:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205074.948964:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188036352:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205074.948974:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205074.948979:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205074/real 1788205074] req@ffff8e82894e7480 x1875062188036352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205074.948985:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7480 x1875062188036352/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205074.948989:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205074.948991:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205074.948992:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188036352:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205074.948995:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188036480:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205074.949003:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205074.949008:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205074/real 1788205074] req@ffff8e82894e4e00 x1875062188036480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205074.949014:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4e00 x1875062188036480/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205074.949018:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205074.949020:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205074.949021:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188036480:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788205075.652532:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205076.676527:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205077.700451:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205078.724504:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205079.748510:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788205080.068449:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205080.068461:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788205080.068466:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205080.068479:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6535 00000100:00080000:2.0:1788205080.068490:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205080.068498:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:2.0:1788205080.068536:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7100 x1875062188036608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205080.068563:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205080.068566:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788205080.068567:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205080.068570:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6535 00000100:00080000:2.0:1788205080.068573:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205080.068576:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205080.068587:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205080.068600:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e4a80 x1875062188036736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205080.068611:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205080.068614:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788205080.068616:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205080.068620:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6535 00000100:00080000:2.0:1788205080.068623:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788205080.068628:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188036608:192.168.201.121@tcp:250:kworker.0 00000100:00080000:2.0:1788205080.068629:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205080.068637:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788205080.068649:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6680 x1875062188036864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205080.068657:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205080.068660:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788205080.068661:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205080.068664:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6535 00000100:00080000:2.0:1788205080.068666:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205080.068670:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205080.068673:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788205080.068680:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e7800 x1875062188036992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788205080.068686:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788205080.068689:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788205080.068690:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788205080.068692:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6535 00000100:00080000:2.0:1788205080.068695:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788205080.068698:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:2.0:1788205080.068701:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788205080.068708:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e82894e6300 x1875062188037120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788205080.068714:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205080.068857:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205080/real 1788205080] req@ffff8e82894e7100 x1875062188036608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205080.068879:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7100 x1875062188036608/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205080.068892:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205080.068896:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205080.068900:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188036608:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788205080.068909:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188036736:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788205080.068935:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205080.069069:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205080/real 1788205080] req@ffff8e82894e4a80 x1875062188036736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205080.069086:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e4a80 x1875062188036736/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205080.069095:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205080.069099:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205080.069140:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188036736:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788205080.069169:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188036864:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788205080.069192:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205080.069205:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205080/real 1788205080] req@ffff8e82894e6680 x1875062188036864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205080.069216:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6680 x1875062188036864/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205080.069224:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205080.069227:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205080.069230:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188036864:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788205080.069235:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188036992:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788205080.069256:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205080.069267:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205080/real 1788205080] req@ffff8e82894e7800 x1875062188036992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205080.069278:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e7800 x1875062188036992/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205080.069286:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205080.069289:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205080.069292:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188036992:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788205080.069340:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188037120:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788205080.069365:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205080.069378:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205080/real 1788205080] req@ffff8e82894e6300 x1875062188037120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205080.069391:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e82894e6300 x1875062188037120/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205080.069399:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205080.069402:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205080.069405:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e82894e6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188037120:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788205080.772681:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205081.796533:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205082.820517:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205083.844540:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205084.868540:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205085.188445:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205085.188454:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205085.188458:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205085.188467:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6540 00000100:00080000:0.0:1788205085.188476:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205085.188483:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788205085.188523:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062188037248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205085.188548:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205085.188551:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205085.188553:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205085.188557:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6540 00000100:00080000:0.0:1788205085.188561:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205085.188566:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205085.188579:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205085.188594:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062188037376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205085.188594:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188037248:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788205085.188604:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205085.188607:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205085.188608:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205085.188612:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6540 00000100:00080000:0.0:1788205085.188615:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205085.188628:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205085.188632:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205085.188642:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062188037504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205085.188649:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205085.188665:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205085.188666:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205085.188669:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6540 00000100:00080000:0.0:1788205085.188672:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788205085.188673:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788205085.188675:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205085.188680:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205085.188688:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062188037632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205085.188696:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205085.188698:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205085.188700:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205085.188703:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6540 00000100:00080000:0.0:1788205085.188706:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205085.188709:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205085.188713:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205085.188723:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062188037760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788205085.188726:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205085/real 1788205085] req@ffff8e828793d500 x1875062188037248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205085.188751:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062188037248/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205085.188766:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205085.188771:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205085.188774:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188037248:192.168.201.121@tcp:250:kworker.0 00000100:00100000:3.0:1788205085.188792:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188037376:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788205085.188808:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205085.188818:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205085/real 1788205085] req@ffff8e828793fb80 x1875062188037376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205085.188829:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062188037376/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205085.188836:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205085.188839:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205085.188842:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188037376:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788205085.188847:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188037504:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788205085.188865:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205085.188876:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205085/real 1788205085] req@ffff8e828793d880 x1875062188037504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205085.188887:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062188037504/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205085.188894:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205085.188897:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205085.188901:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188037504:192.168.201.121@tcp:38:kworker.0 00000100:00100000:3.0:1788205085.188907:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188037632:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788205085.188926:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205085.188936:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205085/real 1788205085] req@ffff8e828793e300 x1875062188037632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205085.188945:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062188037632/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205085.188952:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205085.188955:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205085.188957:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188037632:192.168.201.121@tcp:8:kworker.0 00000100:00100000:3.0:1788205085.188962:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188037760:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788205085.188979:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:3.0:1788205085.188990:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205085/real 1788205085] req@ffff8e828793c380 x1875062188037760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205085.189000:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062188037760/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205085.189007:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205085.189010:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:3.0:1788205085.189013:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188037760:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788205085.892497:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205086.916490:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205087.940465:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205088.964474:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205089.988447:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205090.308415:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205090.308422:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205090.308426:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205090.308437:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6545 00000100:00080000:3.0:1788205090.308445:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205090.308451:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788205090.308480:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062188037888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205090.308502:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205090.308505:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205090.308506:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205090.308508:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6545 00000100:00080000:3.0:1788205090.308511:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205090.308513:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205090.308518:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205090.308525:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062188038016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205090.308532:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205090.308534:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788205090.308535:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205090.308537:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6545 00000100:00080000:3.0:1788205090.308538:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205090.308540:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205090.308543:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205090.308548:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062188038144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205090.308552:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188037888:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788205090.308588:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205090.308590:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788205090.308592:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205090.308595:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6545 00000100:00080000:3.0:1788205090.308597:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205090.308600:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205090.308604:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788205090.308614:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062188038272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205090.308619:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205090.308621:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788205090.308623:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788205090.308627:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788205090.308629:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6545 00000100:00080000:3.0:1788205090.308633:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205090.308635:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205090.308638:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788205090.308645:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062188038400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205090.308671:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205090/real 1788205090] req@ffff8e828953b100 x1875062188037888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205090.308690:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062188037888/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205090.308700:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205090.308704:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205090.308708:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188037888:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788205090.308715:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188038016:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205090.308734:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205090.308750:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205090/real 1788205090] req@ffff8e8289538a80 x1875062188038016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205090.308758:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062188038016/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205090.308763:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205090.308765:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205090.308767:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188038016:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205090.308784:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188038144:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205090.308795:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205090.308801:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205090/real 1788205090] req@ffff8e828953bb80 x1875062188038144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205090.308807:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062188038144/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205090.308812:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205090.308814:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205090.308815:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188038144:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205090.308818:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188038272:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205090.308827:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205090.308832:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205090/real 1788205090] req@ffff8e8289539880 x1875062188038272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205090.308839:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062188038272/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205090.308843:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205090.308844:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205090.308846:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188038272:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205090.308849:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188038400:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205090.308857:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205090.308862:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205090/real 1788205090] req@ffff8e8289539500 x1875062188038400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205090.308867:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062188038400/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205090.308871:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205090.308872:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205090.308874:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188038400:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788205091.012700:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205092.036478:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205093.060509:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205094.084563:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205095.108512:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205095.428431:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205095.428439:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205095.428442:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205095.428457:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6550 00000100:00080000:0.0:1788205095.428466:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205095.428472:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788205095.428516:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062188038528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205095.428545:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205095.428549:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205095.428551:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205095.428556:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6550 00000100:00080000:0.0:1788205095.428560:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205095.428565:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205095.428575:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205095.428586:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188038528:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788205095.428593:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062188038656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205095.428605:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205095.428609:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205095.428611:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205095.428614:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6550 00000100:00080000:0.0:1788205095.428634:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205095.428637:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205095.428641:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788205095.428646:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205095.428653:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062188038784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205095.428664:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205095.428667:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205095.428668:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205095.428672:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6550 00000100:00080000:0.0:1788205095.428677:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205095.428681:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205095.428686:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788205095.428687:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205095/real 1788205095] req@ffff8e828793f480 x1875062188038528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205095.428703:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062188038912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205095.428710:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062188038528/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205095.428713:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205095.428720:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205095.428723:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788205095.428727:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205095.428728:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6550 00000100:00080000:1.0:1788205095.428732:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:0.0:1788205095.428733:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788205095.428736:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188038528:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788205095.428740:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205095.428747:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788205095.428757:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188038656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788205095.428760:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062188039040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205095.428777:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205095.428787:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205095/real 1788205095] req@ffff8e828793dc00 x1875062188038656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205095.428795:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062188038656/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205095.428800:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205095.428807:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205095.428809:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188038656:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205095.428812:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188038784:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205095.428823:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205095.428828:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205095/real 1788205095] req@ffff8e828793c000 x1875062188038784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205095.428835:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062188038784/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205095.428850:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205095.428852:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205095.428854:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188038784:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205095.428857:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188038912:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205095.428868:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205095.428873:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205095/real 1788205095] req@ffff8e828793ca80 x1875062188038912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205095.428879:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062188038912/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205095.428884:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205095.428885:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205095.428888:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188038912:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205095.428901:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188039040:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205095.428929:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205095.428939:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205095/real 1788205095] req@ffff8e828793c700 x1875062188039040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205095.428946:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062188039040/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205095.428950:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205095.428952:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205095.428954:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188039040:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788205096.132502:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205097.156597:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205098.180548:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205099.204673:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205100.228510:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205100.548503:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205100.548518:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205100.548524:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205100.548537:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6555 00000100:00080000:0.0:1788205100.548551:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205100.548561:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788205100.548611:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062188039168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205100.548834:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205100.548838:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205100.548840:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205100.548846:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6555 00000100:00080000:0.0:1788205100.548851:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205100.548856:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205100.548868:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205100.548884:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062188039296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205100.548895:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205100.548898:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205100.548900:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205100.548904:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6555 00000100:00080000:0.0:1788205100.548907:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205100.548910:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205100.548916:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205100.548926:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062188039424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205100.548936:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205100.548939:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205100.548941:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205100.548946:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6555 00000100:00080000:0.0:1788205100.548949:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205100.548953:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205100.548960:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788205100.548965:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188039168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788205100.548975:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062188039552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205100.548986:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205100.548989:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205100.548991:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205100.548995:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6555 00000100:00080000:0.0:1788205100.548999:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205100.549003:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205100.549008:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205100.549023:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062188039680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205100.549176:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188039296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205100.549207:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188039424:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205100.549247:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188039552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205100.549273:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188039680:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788205101.252900:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205102.283086:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205103.300658:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788205103.620635:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:2.0:1788205103.620679:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788205103.620695:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788205103.620735:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788205103.620745:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788205103.620753:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000800:00000100:2.0:1788205103.620762:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788205103.620830:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205100/real 1788205103] req@ffff8e828793f100 x1875062188039552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205103.620871:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062188039552/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205103.620885:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205103.620890:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205103.620895:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188039552:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205103.620903:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205100/real 1788205103] req@ffff8e828793c700 x1875062188039680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205103.620914:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062188039680/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205103.620919:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205103.620922:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205103.620924:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188039680:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205103.620928:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205100/real 1788205103] req@ffff8e828793ce00 x1875062188039168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205103.620935:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062188039168/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205103.620939:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205103.620941:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205103.620943:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188039168:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788205103.620950:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205100/real 1788205103] req@ffff8e828793f800 x1875062188039296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205103.620961:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062188039296/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205103.620969:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205103.620972:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205103.620975:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188039296:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205103.620981:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205100/real 1788205103] req@ffff8e828793e680 x1875062188039424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205103.620992:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062188039424/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205103.621000:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205103.621003:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205103.621007:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188039424:192.168.201.121@tcp:38:kworker.0 00010000:00010000:0.0:1788205104.325051:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205105.348563:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205105.668535:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205105.668546:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205105.668550:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205105.668559:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6560 00000100:00080000:3.0:1788205105.668570:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205105.668576:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788205105.668613:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062188039808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205105.668642:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205105.668646:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205105.668648:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205105.668652:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6560 00000100:00080000:3.0:1788205105.668656:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205105.668660:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205105.668669:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205105.668683:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062188039936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205105.668692:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205105.668695:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788205105.668696:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205105.668699:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6560 00000100:00080000:3.0:1788205105.668702:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205105.668705:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205105.668709:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205105.668717:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062188040064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205105.668724:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205105.668727:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788205105.668729:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205105.668731:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6560 00000100:00080000:3.0:1788205105.668734:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205105.668737:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205105.668741:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788205105.668749:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062188040192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205105.668758:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205105.668761:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788205105.668762:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205105.668765:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6560 00000100:00080000:3.0:1788205105.668768:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205105.668771:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205105.668774:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788205105.668782:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062188040320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205105.668856:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188039808:192.168.201.121@tcp:250:kworker.0 00000800:00000100:1.0:1788205105.668912:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205105.669013:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205105/real 1788205105] req@ffff8e8289538380 x1875062188039808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205105.669030:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062188039808/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205105.669039:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205105.669043:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205105.669047:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188039808:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788205105.669053:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188039936:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205105.669075:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205105.669086:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205105/real 1788205105] req@ffff8e8289539f80 x1875062188039936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205105.669096:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062188039936/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205105.669103:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205105.669105:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205105.669108:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188039936:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205105.669114:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188040064:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205105.669133:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205105.669145:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205105/real 1788205105] req@ffff8e828953a680 x1875062188040064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205105.669156:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062188040064/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205105.669164:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205105.669167:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205105.669171:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188040064:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205105.669176:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188040192:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205105.669211:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205105.669221:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205105/real 1788205105] req@ffff8e8289539500 x1875062188040192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205105.669231:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062188040192/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205105.669239:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205105.669242:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205105.669245:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188040192:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205105.669251:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188040320:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205105.669270:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205105.669279:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205105/real 1788205105] req@ffff8e8289539880 x1875062188040320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205105.669345:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062188040320/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205105.669354:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205105.669357:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205105.669360:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188040320:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788205106.372490:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205107.398877:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205108.421341:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205109.447349:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205110.468494:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205110.788665:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205110.788674:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205110.788678:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205110.788688:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6565 00000100:00080000:0.0:1788205110.788698:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205110.788704:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788205110.788736:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062188040448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205110.788768:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205110.788771:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205110.788773:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205110.788777:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6565 00000100:00080000:0.0:1788205110.788780:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205110.788783:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205110.788792:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205110.788803:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062188040576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205110.788812:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205110.788815:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205110.788817:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205110.788820:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6565 00000100:00080000:0.0:1788205110.788822:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205110.788826:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205110.788830:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205110.788840:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062188040704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205110.788848:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205110.788851:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205110.788853:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205110.788856:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6565 00000100:00080000:0.0:1788205110.788859:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205110.788862:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205110.788866:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205110.788875:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062188040832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205110.788884:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205110.788886:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205110.788889:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205110.788892:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6565 00000100:00080000:0.0:1788205110.788894:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205110.788898:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205110.788902:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205110.788911:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062188040960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205110.789008:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188040448:192.168.201.121@tcp:250:kworker.0 00000800:00000100:1.0:1788205110.789066:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205110.789102:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205110/real 1788205110] req@ffff8e828793ca80 x1875062188040448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205110.789117:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062188040448/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205110.789127:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205110.789131:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205110.789134:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188040448:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788205110.789141:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188040576:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205110.789160:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205110.789171:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205110/real 1788205110] req@ffff8e828793c000 x1875062188040576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205110.789182:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062188040576/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205110.789190:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205110.789193:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205110.789196:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188040576:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205110.789201:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188040704:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205110.789234:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205110.789245:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205110/real 1788205110] req@ffff8e828793dc00 x1875062188040704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205110.789255:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062188040704/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205110.789263:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205110.789266:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205110.789268:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188040704:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205110.789274:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188040832:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205110.789290:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205110.789298:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205110/real 1788205110] req@ffff8e828793f480 x1875062188040832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205110.789332:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062188040832/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205110.789339:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205110.789342:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205110.789346:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188040832:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205110.789351:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188040960:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205110.789370:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205110.789380:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205110/real 1788205110] req@ffff8e828793c380 x1875062188040960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205110.789391:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062188040960/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205110.789399:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205110.789401:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205110.789404:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188040960:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788205111.492834:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205112.516550:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205113.541726:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205114.566210:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205115.588501:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205115.910267:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205115.910276:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205115.910280:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205115.910290:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6570 00000100:00080000:3.0:1788205115.910300:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205115.910330:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788205115.910365:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062188041088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205115.910398:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205115.910402:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205115.910404:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205115.910408:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6570 00000100:00080000:3.0:1788205115.910412:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205115.910416:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205115.910425:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205115.910439:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062188041216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205115.910448:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205115.910451:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788205115.910453:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205115.910457:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6570 00000100:00100000:0.0:1788205115.910464:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188041088:192.168.201.121@tcp:250:kworker.0 00000800:00000100:0.0:1788205115.910525:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788205115.910564:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788205115.910569:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205115/real 1788205115] req@ffff8e828953bb80 x1875062188041088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205115.910573:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205115.910581:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205115.910593:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062188041088/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205115.910598:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062188041344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205115.910614:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788205115.910619:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205115.910623:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00080000:3.0:1788205115.910626:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788205115.910630:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788205115.910631:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188041088:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788205115.910638:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6570 00000100:00080000:3.0:1788205115.910644:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788205115.910645:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188041216:192.168.201.121@tcp:38:kworker.0 00000100:00080000:3.0:1788205115.910651:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205115.910658:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788205115.910672:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062188041472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205115.910678:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788205115.910687:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205115.910691:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788205115.910692:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788205115.910698:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205115/real 1788205115] req@ffff8e8289538a80 x1875062188041216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788205115.910698:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6570 00000100:00080000:3.0:1788205115.910703:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205115.910711:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205115.910720:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205115.910727:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062188041216/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788205115.910736:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062188041600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205115.910738:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205115.910809:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205115.910814:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188041216:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788205115.910841:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188041344:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788205115.910866:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205115.910883:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205115/real 1788205115] req@ffff8e828953b100 x1875062188041344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205115.910897:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062188041344/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205115.910907:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205115.910911:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205115.910916:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188041344:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788205115.910923:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188041472:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788205115.910942:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205115.910952:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205115/real 1788205115] req@ffff8e8289539880 x1875062188041472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205115.910963:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062188041472/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205115.910976:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205115.910980:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205115.910983:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188041472:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788205115.910989:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188041600:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788205115.912116:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205115.912141:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205115/real 1788205115] req@ffff8e8289539500 x1875062188041600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205115.912155:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062188041600/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205115.912164:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205115.912167:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205115.912170:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188041600:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788205116.612501:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205117.636526:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205118.660671:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205119.685364:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205120.708612:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205121.028439:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205121.028448:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205121.028451:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205121.028466:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6576 00000100:00080000:3.0:1788205121.028475:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205121.028481:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788205121.028513:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062188041728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205121.028545:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205121.028551:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205121.028552:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205121.028556:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6576 00000100:00080000:3.0:1788205121.028559:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205121.028562:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205121.028570:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205121.028582:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062188041856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205121.028592:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205121.028596:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788205121.028597:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205121.028603:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6576 00000100:00080000:3.0:1788205121.028606:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205121.028609:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205121.028613:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205121.028622:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062188041984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205121.028633:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205121.028637:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788205121.028639:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188041728:192.168.201.121@tcp:250:kworker.0 00000100:00080000:3.0:1788205121.028640:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205121.028650:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6576 00000100:00080000:3.0:1788205121.028656:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205121.028660:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205121.028665:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788205121.028676:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062188042112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205121.028684:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205121.028687:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788205121.028689:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205121.028692:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6576 00000100:00080000:3.0:1788205121.028695:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205121.028698:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205121.028704:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788205121.028713:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062188042240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788205121.028736:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205121.028807:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205121/real 1788205121] req@ffff8e828953a680 x1875062188041728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205121.028827:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062188041728/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205121.028844:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205121.028848:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205121.028852:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188041728:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788205121.028862:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188041856:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788205121.028887:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205121.028899:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205121/real 1788205121] req@ffff8e8289539f80 x1875062188041856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205121.028910:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062188041856/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205121.028918:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205121.028921:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205121.028925:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188041856:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788205121.028954:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188041984:192.168.201.121@tcp:38:kworker.0 00000800:00000100:0.0:1788205121.028975:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205121.028986:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205121/real 1788205121] req@ffff8e8289538380 x1875062188041984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205121.029007:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062188041984/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205121.029015:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205121.029019:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205121.029022:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188041984:192.168.201.121@tcp:38:kworker.0 00000100:00100000:0.0:1788205121.029029:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188042112:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788205121.029074:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205121.029085:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205121/real 1788205121] req@ffff8e8289539500 x1875062188042112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205121.029096:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062188042112/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205121.029105:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205121.029108:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205121.029111:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188042112:192.168.201.121@tcp:8:kworker.0 00000100:00100000:0.0:1788205121.029118:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188042240:192.168.201.121@tcp:8:kworker.0 00000800:00000100:0.0:1788205121.029137:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205121.029224:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205121/real 1788205121] req@ffff8e8289539880 x1875062188042240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205121.029237:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062188042240/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788205121.029245:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788205121.029249:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:0.0:1788205121.029259:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188042240:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788205121.732522:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205122.756457:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205123.780514:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205124.804484:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205125.828543:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205126.148520:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205126.148529:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205126.148533:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205126.148541:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6581 00000100:00080000:0.0:1788205126.148553:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205126.148559:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788205126.148594:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062188042368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205126.148622:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205126.148626:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205126.148628:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205126.148632:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6581 00000100:00080000:0.0:1788205126.148635:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205126.148639:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205126.148648:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205126.148660:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062188042496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205126.148668:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205126.148671:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788205126.148672:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188042368:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788205126.148673:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205126.148681:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6581 00000100:00080000:0.0:1788205126.148685:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205126.148688:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205126.148693:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205126.148704:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062188042624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205126.148715:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205126.148719:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205126.148720:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205126.148724:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6581 00000100:00080000:0.0:1788205126.148729:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205126.148733:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205126.148738:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788205126.148740:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205126.148755:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062188042752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205126.148765:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205126.148768:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205126.148770:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205126.148774:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6581 00000100:00080000:0.0:1788205126.148777:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205126.148781:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788205126.148785:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205126/real 1788205126] req@ffff8e828793e300 x1875062188042368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788205126.148788:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205126.148806:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062188042880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205126.148810:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062188042368/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205126.148832:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205126.148836:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205126.148840:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188042368:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788205126.148864:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188042496:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205126.148884:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205126.148895:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205126/real 1788205126] req@ffff8e828793d880 x1875062188042496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205126.148906:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062188042496/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205126.148926:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205126.148929:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205126.148932:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188042496:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205126.148938:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188042624:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205126.148984:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205126.148995:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205126/real 1788205126] req@ffff8e828793fb80 x1875062188042624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205126.149005:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062188042624/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205126.149017:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205126.149020:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205126.149024:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188042624:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205126.149030:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188042752:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205126.149049:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205126.149059:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205126/real 1788205126] req@ffff8e828793d500 x1875062188042752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205126.149069:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062188042752/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205126.149076:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205126.149078:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205126.149089:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188042752:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205126.149094:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188042880:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205126.149111:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205126.149120:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205126/real 1788205126] req@ffff8e828793c380 x1875062188042880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205126.149130:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062188042880/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205126.149137:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205126.149140:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205126.149143:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188042880:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788205126.852488:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205127.876452:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205128.900492:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205129.924442:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205130.948437:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205131.268395:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205131.268404:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205131.268408:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205131.268416:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6586 00000100:00080000:0.0:1788205131.268424:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205131.268429:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788205131.268461:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062188043008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205131.268481:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205131.268484:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205131.268485:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205131.268488:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6586 00000100:00080000:0.0:1788205131.268490:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205131.268492:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205131.268499:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205131.268507:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062188043136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205131.268514:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205131.268516:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205131.268517:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205131.268519:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6586 00000100:00080000:0.0:1788205131.268521:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205131.268526:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788205131.268528:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188043008:192.168.201.121@tcp:250:kworker.0 00000008:00100000:0.0:1788205131.268529:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205131.268540:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062188043264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205131.268550:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205131.268552:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205131.268554:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205131.268556:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6586 00000100:00080000:0.0:1788205131.268558:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205131.268560:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205131.268563:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205131.268569:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062188043392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205131.268575:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205131.268577:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205131.268578:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205131.268580:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6586 00000100:00080000:0.0:1788205131.268582:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788205131.268585:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788205131.268587:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205131.268590:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205131.268597:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062188043520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205131.268620:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205131/real 1788205131] req@ffff8e828793f480 x1875062188043008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205131.268633:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062188043008/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205131.268644:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205131.268647:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205131.268649:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188043008:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788205131.268654:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188043136:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205131.268666:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205131.268673:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205131/real 1788205131] req@ffff8e828793dc00 x1875062188043136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205131.268679:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062188043136/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205131.268684:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205131.268686:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205131.268688:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188043136:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205131.268711:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188043264:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205131.268722:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205131.268728:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205131/real 1788205131] req@ffff8e828793c000 x1875062188043264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205131.268735:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062188043264/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205131.268739:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205131.268741:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205131.268742:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188043264:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205131.268746:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188043392:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205131.268755:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205131.268760:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205131/real 1788205131] req@ffff8e828793ca80 x1875062188043392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205131.268766:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062188043392/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205131.268771:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205131.268772:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205131.268774:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188043392:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205131.268776:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188043520:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205131.268784:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205131.268789:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205131/real 1788205131] req@ffff8e828793e680 x1875062188043520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205131.268795:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062188043520/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205131.268800:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205131.268801:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205131.268803:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188043520:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788205131.972974:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205132.996546:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205134.020514:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205135.044541:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205136.068899:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205136.388453:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205136.388466:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205136.388471:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205136.388484:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6591 00000100:00080000:0.0:1788205136.388497:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205136.388505:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788205136.388552:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062188043648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205136.388587:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205136.388591:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205136.388592:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205136.388597:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6591 00000100:00080000:0.0:1788205136.388600:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205136.388604:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205136.388614:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205136.388625:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062188043776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205136.388633:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205136.388637:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205136.388638:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205136.388641:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6591 00000100:00080000:0.0:1788205136.388644:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205136.388647:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205136.388651:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205136.388658:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062188043904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205136.388667:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205136.388670:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205136.388671:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205136.388674:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6591 00000100:00080000:0.0:1788205136.388676:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205136.388679:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205136.388683:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205136.388691:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062188044032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205136.388698:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205136.388701:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205136.388702:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205136.388705:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6591 00000100:00080000:0.0:1788205136.388707:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205136.388710:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205136.388714:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205136.388722:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062188044160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205136.388831:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188043648:192.168.201.121@tcp:250:kworker.0 00000800:00000100:1.0:1788205136.388904:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205136.388962:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205136/real 1788205136] req@ffff8e828793f800 x1875062188043648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205136.388981:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062188043648/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205136.388992:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205136.388995:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205136.388999:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188043648:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788205136.389007:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188043776:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205136.389027:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205136.389039:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205136/real 1788205136] req@ffff8e828793ce00 x1875062188043776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205136.389050:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062188043776/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205136.389058:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205136.389060:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205136.389064:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188043776:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205136.389069:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188043904:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205136.389086:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205136.389095:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205136/real 1788205136] req@ffff8e828793c700 x1875062188043904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205136.389106:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062188043904/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205136.389119:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205136.389122:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205136.389124:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188043904:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205136.389129:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188044032:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205136.389146:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205136.389156:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205136/real 1788205136] req@ffff8e828793f100 x1875062188044032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205136.389167:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062188044032/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205136.389174:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205136.389178:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205136.389181:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188044032:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205136.389186:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188044160:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205136.389206:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205136.389217:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205136/real 1788205136] req@ffff8e828793e680 x1875062188044160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205136.389229:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062188044160/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205136.389236:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205136.389239:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205136.389243:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188044160:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788205137.092465:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205138.116519:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205139.140647:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205140.164456:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205141.188518:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205141.508506:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205141.508518:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205141.508523:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205141.508533:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6596 00000100:00080000:3.0:1788205141.508543:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205141.508560:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788205141.508610:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953b100 x1875062188044288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205141.508643:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205141.508648:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205141.508650:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205141.508654:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6596 00000100:00080000:3.0:1788205141.508658:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205141.508662:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205141.508686:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205141.508702:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538a80 x1875062188044416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205141.508711:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205141.508714:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788205141.508716:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205141.508720:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6596 00000100:00080000:3.0:1788205141.508723:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205141.508727:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205141.508731:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205141.508741:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953bb80 x1875062188044544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205141.508750:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205141.508753:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788205141.508755:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205141.508758:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6596 00000100:00080000:3.0:1788205141.508762:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205141.508766:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205141.508770:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788205141.508780:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539880 x1875062188044672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205141.508795:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205141.508799:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788205141.508801:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205141.508804:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6596 00000100:00080000:3.0:1788205141.508807:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205141.508811:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:1.0:1788205141.509061:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188044288:192.168.201.121@tcp:250:kworker.0 00000800:00000100:1.0:1788205141.509129:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205141.509226:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205141/real 1788205141] req@ffff8e828953b100 x1875062188044288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205141.509245:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953b100 x1875062188044288/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205141.509256:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205141.509260:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205141.509264:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188044288:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788205141.509271:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188044416:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205141.509294:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205141.509333:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205141/real 1788205141] req@ffff8e8289538a80 x1875062188044416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205141.509347:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538a80 x1875062188044416/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205141.509355:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205141.509358:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205141.509362:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188044416:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205141.509368:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188044544:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205141.509395:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205141.509407:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205141/real 1788205141] req@ffff8e828953bb80 x1875062188044544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205141.509419:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953bb80 x1875062188044544/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205141.509427:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205141.509430:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205141.509433:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188044544:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205141.509439:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188044672:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205141.509458:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205141.509469:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205141/real 1788205141] req@ffff8e8289539880 x1875062188044672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205141.509480:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539880 x1875062188044672/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205141.509487:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205141.509490:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205141.509493:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188044672:192.168.201.121@tcp:8:kworker.0 00000008:00100000:3.0:1788205141.509744:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788205141.509761:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539500 x1875062188044800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205141.509831:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188044800:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205141.509858:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205141.511605:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205141/real 1788205141] req@ffff8e8289539500 x1875062188044800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205141.511621:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539500 x1875062188044800/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205141.511629:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205141.511631:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205141.511634:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188044800:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788205142.213016:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205143.236456:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205144.260566:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205145.284872:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205146.308463:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788205146.628728:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205146.628738:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788205146.628742:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205146.628752:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6601 00000100:00080000:3.0:1788205146.628762:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205146.628768:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:3.0:1788205146.628801:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062188044928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205146.628871:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188044928:192.168.201.121@tcp:250:kworker.0 00000800:00000100:1.0:1788205146.628926:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:3.0:1788205146.629563:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205146.629567:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788205146.629569:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205146.629574:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6601 00000100:00080000:3.0:1788205146.629578:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205146.629582:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205146.629599:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205146.629617:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062188045056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205146.632342:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205146/real 1788205146] req@ffff8e8289538380 x1875062188044928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205146.632361:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062188044928/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205146.632368:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205146.632371:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205146.632374:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188044928:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788205146.632396:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188045056:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205146.632434:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205146.632444:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205146/real 1788205146] req@ffff8e8289539f80 x1875062188045056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205146.632451:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062188045056/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205146.632456:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205146.632458:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205146.632460:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188045056:192.168.201.121@tcp:38:kworker.0 00000100:00080000:3.0:1788205146.632808:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205146.632814:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788205146.632817:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205146.632822:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6601 00000100:00080000:3.0:1788205146.632827:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205146.632833:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205146.632841:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788205146.632863:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828953a680 x1875062188045184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205146.632876:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205146.632879:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788205146.632881:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205146.632884:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6601 00000100:00080000:3.0:1788205146.632887:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205146.632890:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205146.632894:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788205146.632903:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289539f80 x1875062188045312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788205146.632912:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788205146.632915:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788205146.632917:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788205146.632920:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6601 00000100:00080000:3.0:1788205146.632923:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788205146.632926:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:3.0:1788205146.632930:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788205146.632938:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e8289538380 x1875062188045440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205146.633063:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188045184:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205146.633089:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205146.633102:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205146/real 1788205146] req@ffff8e828953a680 x1875062188045184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205146.633114:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828953a680 x1875062188045184/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205146.633123:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205146.633126:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205146.633129:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828953a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188045184:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205146.633135:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188045312:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205146.633151:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205146.633159:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205146/real 1788205146] req@ffff8e8289539f80 x1875062188045312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205146.633170:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289539f80 x1875062188045312/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205146.633177:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205146.633180:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205146.633183:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289539f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188045312:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205146.633188:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188045440:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205146.633202:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205146.633210:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205146/real 1788205146] req@ffff8e8289538380 x1875062188045440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205146.633220:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e8289538380 x1875062188045440/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205146.633228:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205146.633230:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205146.633233:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e8289538380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188045440:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788205147.332778:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205148.357701:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205149.380477:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205150.404496:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205151.428516:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205151.748394:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205151.748409:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205151.748414:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205151.748427:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6606 00000100:00080000:0.0:1788205151.748439:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205151.748448:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788205151.748496:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062188045568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205151.748535:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205151.748538:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205151.748539:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205151.748543:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6606 00000100:00080000:0.0:1788205151.748546:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205151.748549:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205151.748558:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205151.748569:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062188045696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205151.748576:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205151.748578:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205151.748579:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205151.748581:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6606 00000100:00080000:0.0:1788205151.748584:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205151.748587:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205151.748590:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788205151.748596:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188045568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788205151.748599:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062188045824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205151.748609:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205151.748611:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205151.748612:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205151.748615:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6606 00000100:00080000:0.0:1788205151.748616:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205151.748618:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205151.748621:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205151.748628:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062188045952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205151.748633:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205151.748635:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205151.748637:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205151.748639:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6606 00000100:00080000:0.0:1788205151.748641:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205151.748643:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205151.748645:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205151.748652:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062188046080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205151.748690:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205151.748744:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205151/real 1788205151] req@ffff8e828793ca80 x1875062188045568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205151.748765:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062188045568/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205151.748778:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205151.748782:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205151.748785:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188045568:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788205151.748791:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188045696:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205151.748817:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205151.748831:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205151/real 1788205151] req@ffff8e828793c000 x1875062188045696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205151.748843:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062188045696/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205151.748858:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205151.748862:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205151.748866:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188045696:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205151.748901:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188045824:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205151.748926:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205151.748939:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205151/real 1788205151] req@ffff8e828793dc00 x1875062188045824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205151.748950:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062188045824/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205151.748957:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205151.748960:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205151.748963:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188045824:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205151.748969:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188045952:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205151.748991:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205151.749005:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205151/real 1788205151] req@ffff8e828793f480 x1875062188045952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205151.749013:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062188045952/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205151.749018:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205151.749020:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205151.749022:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188045952:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205151.749026:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188046080:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205151.749039:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205151.749046:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205151/real 1788205151] req@ffff8e828793c380 x1875062188046080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205151.749054:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062188046080/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205151.749067:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205151.749070:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205151.749073:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188046080:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788205152.452517:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205153.476542:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205154.500424:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205155.524473:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205156.548544:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205156.868492:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205156.868506:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205156.868511:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205156.868523:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6611 00000100:00080000:0.0:1788205156.868537:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205156.868546:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788205156.868594:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d500 x1875062188046208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205156.868627:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205156.868630:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205156.868632:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205156.868637:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6611 00000100:00080000:0.0:1788205156.868640:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205156.868645:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205156.868656:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205156.868671:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793fb80 x1875062188046336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205156.868680:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188046208:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788205156.868685:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205156.868690:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205156.868692:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205156.868696:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6611 00000100:00080000:0.0:1788205156.868701:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205156.868705:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205156.868710:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205156.868723:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793d880 x1875062188046464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205156.868733:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205156.868736:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205156.868738:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205156.868743:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6611 00000100:00080000:0.0:1788205156.868747:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205156.868751:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205156.868756:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788205156.868760:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:0.0:1788205156.868771:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e300 x1875062188046592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205156.868781:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205156.868784:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205156.868786:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205156.868791:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6611 00000100:00080000:0.0:1788205156.868795:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205156.868799:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205156.868804:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788205156.868814:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205156/real 1788205156] req@ffff8e828793d500 x1875062188046208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205156.868819:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062188046720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205156.868838:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d500 x1875062188046208/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205156.868848:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205156.868851:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205156.868855:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188046208:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788205156.868883:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188046336:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205156.868908:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205156.868917:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205156/real 1788205156] req@ffff8e828793fb80 x1875062188046336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205156.868928:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793fb80 x1875062188046336/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205156.868936:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205156.868939:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205156.868942:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188046336:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205156.868948:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188046464:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205156.868968:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205156.868979:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205156/real 1788205156] req@ffff8e828793d880 x1875062188046464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205156.868990:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793d880 x1875062188046464/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205156.868998:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205156.869001:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205156.869004:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188046464:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205156.869010:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188046592:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205156.869026:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205156.869032:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205156/real 1788205156] req@ffff8e828793e300 x1875062188046592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205156.869044:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e300 x1875062188046592/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205156.869049:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205156.869050:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205156.869052:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188046592:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205156.869055:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188046720:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205156.869064:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205156.869069:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205156/real 1788205156] req@ffff8e828793c380 x1875062188046720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205156.869075:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062188046720/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205156.869080:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205156.869081:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205156.869083:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188046720:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788205157.572543:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205158.596629:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205159.620551:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205160.644572:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205161.668543:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205161.988463:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205161.988476:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205161.988480:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205161.988493:0:43172:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6617 00000100:00080000:0.0:1788205161.988506:0:43172:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205161.988515:0:43172:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788205161.988573:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062188046848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205161.988608:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205161.988612:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205161.988621:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205161.988626:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6617 00000100:00080000:0.0:1788205161.988630:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205161.988635:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205161.988646:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205161.988663:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062188046976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205161.988663:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188046848:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788205161.988686:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205161.988690:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205161.988692:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205161.988696:0:43172:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6617 00000100:00080000:0.0:1788205161.988700:0:43172:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205161.988705:0:43172:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205161.988711:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205161.988724:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062188047104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205161.988732:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205161.988736:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205161.988738:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205161.988744:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6617 00000800:00000100:1.0:1788205161.988745:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00080000:0.0:1788205161.988750:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205161.988755:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205161.988761:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205161.988771:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062188047232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205161.988780:0:43172:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205161.988783:0:43172:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205161.988785:0:43172:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205161.988789:0:43172:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6617 00000100:00080000:0.0:1788205161.988793:0:43172:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205161.988797:0:43172:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205161.988802:0:43172:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788205161.988808:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205161/real 1788205161] req@ffff8e828793f480 x1875062188046848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788205161.988815:0:43172:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062188047360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205161.988834:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062188046848/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205161.988846:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205161.988849:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205161.988853:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188046848:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788205161.988877:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188046976:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205161.988895:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205161.988904:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205161/real 1788205161] req@ffff8e828793dc00 x1875062188046976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205161.988912:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062188046976/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205161.988919:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205161.988921:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205161.988923:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188046976:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205161.988927:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188047104:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205161.988939:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205161.988947:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205161/real 1788205161] req@ffff8e828793c000 x1875062188047104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205161.988954:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062188047104/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205161.988959:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205161.988961:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205161.988963:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188047104:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205161.988968:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188047232:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205161.988982:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205161.988988:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205161/real 1788205161] req@ffff8e828793ca80 x1875062188047232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205161.988995:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062188047232/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205161.989002:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205161.989004:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205161.989006:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188047232:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205161.989010:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188047360:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205161.989021:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205161.989028:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205161/real 1788205161] req@ffff8e828793e680 x1875062188047360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205161.989036:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062188047360/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205161.989040:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205161.989042:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205161.989044:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188047360:192.168.201.121@tcp:8:kworker.0 00010000:00010000:0.0:1788205162.692541:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205163.716541:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205164.740406:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205165.765162:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205166.788628:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205167.108391:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205167.108406:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205167.108411:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205167.108424:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6622 00000100:00080000:0.0:1788205167.108448:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205167.108457:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788205167.108503:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f100 x1875062188047488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205167.108536:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205167.108540:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205167.108542:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205167.108546:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6622 00000100:00080000:0.0:1788205167.108550:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205167.108554:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205167.108565:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205167.108579:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c700 x1875062188047616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205167.108591:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205167.108596:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205167.108599:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788205167.108599:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188047488:192.168.201.121@tcp:250:kworker.0 00000100:00080000:0.0:1788205167.108607:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6622 00000100:00080000:0.0:1788205167.108612:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205167.108617:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205167.108624:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205167.108637:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ce00 x1875062188047744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205167.108646:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205167.108650:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205167.108652:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205167.108657:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6622 00000100:00080000:0.0:1788205167.108660:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205167.108666:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205167.108671:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205167.108681:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f800 x1875062188047872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205167.108691:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205167.108694:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205167.108696:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205167.108700:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6622 00000100:00080000:0.0:1788205167.108703:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205167.108707:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205167.108712:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205167.108725:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793e680 x1875062188048000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788205167.108752:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188047616:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205167.108786:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188047744:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205167.108812:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188047872:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205167.108836:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188048000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788205167.812487:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1788205168.836493:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205169.860911:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1788205170.180649:0:2391:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.121:988 00000400:00000100:3.0:1788205170.180694:0:2391:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.121@tcp at host 192.168.201.121 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788205170.180711:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788205170.180887:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205167/real 1788205170] req@ffff8e828793f100 x1875062188047488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205192 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205170.180913:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f100 x1875062188047488/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205192 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205170.180994:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205170.181000:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205170.181005:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188047488:192.168.201.121@tcp:250:kworker.0 00000800:00000100:3.0:1788205170.181071:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788205170.181232:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205167/real 1788205170] req@ffff8e828793c700 x1875062188047616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205192 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205170.181249:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c700 x1875062188047616/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205192 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205170.181258:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205170.181262:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205170.181266:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188047616:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788205170.181352:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788205170.181487:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205167/real 1788205170] req@ffff8e828793ce00 x1875062188047744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205192 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205170.181503:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ce00 x1875062188047744/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205192 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205170.181512:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205170.181516:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205170.181521:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188047744:192.168.201.121@tcp:38:kworker.0 00000800:00000100:3.0:1788205170.181573:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788205170.181646:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205167/real 1788205170] req@ffff8e828793f800 x1875062188047872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205192 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205170.181661:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f800 x1875062188047872/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205192 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205170.181669:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205170.181672:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205170.181675:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188047872:192.168.201.121@tcp:8:kworker.0 00000800:00000100:3.0:1788205170.181715:0:2391:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.121@tcp 00000100:00100000:1.0:1788205170.181780:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205167/real 1788205170] req@ffff8e828793e680 x1875062188048000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205192 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205170.181793:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793e680 x1875062188048000/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205192 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205170.181801:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205170.181804:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205170.181807:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188048000:192.168.201.121@tcp:8:kworker.0 00010000:00010000:2.0:1788205170.884432:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205171.908549:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788205172.228413:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) 4799c7f6-f128-4e9d-a70c-dd4e15fe111d->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205172.228424:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788205172.228428:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205172.228444:0:11298:0:(import.c:510:import_select_connection()) MGC192.168.201.121@tcp: connect to NID 192.168.201.121@tcp last attempt 6627 00000100:00080000:0.0:1788205172.228456:0:11298:0:(import.c:589:import_select_connection()) MGC192.168.201.121@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205172.228461:0:11298:0:(import.c:624:import_select_connection()) MGC192.168.201.121@tcp: import ffff8e8292255800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000100:00100000:0.0:1788205172.228490:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793ca80 x1875062188048128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205172.228512:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205172.228514:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788205172.228515:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205172.228518:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6627 00000100:00080000:0.0:1788205172.228520:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205172.228522:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8e8282a4b800: import ffff8e82871d2800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205172.228529:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205172.228535:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c000 x1875062188048256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205172.228542:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205172.228545:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788205172.228546:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205172.228548:0:11298:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6627 00000100:00080000:0.0:1788205172.228550:0:11298:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205172.228552:0:11298:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff8e8282a4b800: import ffff8e8291097000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205172.228556:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683239 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788205172.228562:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793dc00 x1875062188048384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205172.228569:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205172.228573:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788205172.228578:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205172.228582:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6627 00000100:00080000:0.0:1788205172.228586:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205172.228594:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8e8282a4b800: import ffff8e8291096800 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205172.228599:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788205172.228605:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188048128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:0.0:1788205172.228613:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793f480 x1875062188048512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788205172.228625:0:11298:0:(pinger.c:229:ptlrpc_pinger_process_import()) dc31b02e-cf70-4b83-9f70-d7124af83e4d->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788205172.228629:0:11298:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788205172.228631:0:11298:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788205172.228634:0:11298:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: connect to NID 192.168.201.121@tcp last attempt 6627 00000100:00080000:0.0:1788205172.228639:0:11298:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788205172.228643:0:11298:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8e8282a4b800: import ffff8e8291090000 using connection 192.168.201.121@tcp/192.168.201.121@tcp 00000008:00100000:0.0:1788205172.228648:0:11298:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683239 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788205172.228667:0:11298:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8e828793c380 x1875062188048640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788205172.228702:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205172.228755:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205172/real 1788205172] req@ffff8e828793ca80 x1875062188048128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205172.228775:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793ca80 x1875062188048128/t0(0) o250->MGC192.168.201.121@tcp@192.168.201.121@tcp:26/25 lens 520/544 e 0 to 1 dl 1788205197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205172.228786:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8292255800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205172.228791:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205172.228795:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:4799c7f6-f128-4e9d-a70c-dd4e15fe111d:2400:1875062188048128:192.168.201.121@tcp:250:kworker.0 00000100:00100000:1.0:1788205172.228805:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188048256:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205172.228827:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205172.228838:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205172/real 1788205172] req@ffff8e828793c000 x1875062188048256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205172.228849:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c000 x1875062188048256/t0(0) o38->lustre-MDT0000-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205172.228857:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e82871d2800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205172.228861:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205172.228864:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188048256:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205172.228870:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188048384:192.168.201.121@tcp:38:kworker.0 00000800:00000100:1.0:1788205172.228889:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205172.228900:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205172/real 1788205172] req@ffff8e828793dc00 x1875062188048384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205172.228917:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793dc00 x1875062188048384/t0(0) o38->lustre-MDT0001-mdc-ffff8e8282a4b800@192.168.201.121@tcp:12/10 lens 520/544 e 0 to 1 dl 1788205197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205172.228924:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291097000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205172.228927:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205172.228930:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188048384:192.168.201.121@tcp:38:kworker.0 00000100:00100000:1.0:1788205172.228961:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188048512:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205172.228981:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205172.228992:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205172/real 1788205172] req@ffff8e828793f480 x1875062188048512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205172.229003:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793f480 x1875062188048512/t0(0) o8->lustre-OST0000-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205172.229011:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291096800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205172.229014:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205172.229017:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188048512:192.168.201.121@tcp:8:kworker.0 00000100:00100000:1.0:1788205172.229023:0:2400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188048640:192.168.201.121@tcp:8:kworker.0 00000800:00000100:1.0:1788205172.229040:0:2400:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.121@tcp 00000100:00100000:1.0:1788205172.229050:0:2400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788205172/real 1788205172] req@ffff8e828793c380 x1875062188048640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788205172.229060:0:2400:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8e828793c380 x1875062188048640/t0(0) o8->lustre-OST0001-osc-ffff8e8282a4b800@192.168.201.121@tcp:28/4 lens 520/544 e 0 to 1 dl 1788205197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788205172.229073:0:2400:0:(import.c:65:import_set_state_nolock()) ffff8e8291090000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788205172.229077:0:2400:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.121@tcp failed (-110) 00000100:00100000:1.0:1788205172.229080:0:2400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8e828793c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:dc31b02e-cf70-4b83-9f70-d7124af83e4d:2400:1875062188048640:192.168.201.121@tcp:8:kworker.0 00010000:00010000:1.0:1788205172.932543:0:25294:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788205173.956543:0:6173:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788205174.980962:0:6172:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 72246 lines, 72246 kept, 0 dropped, 0 bad.